I knew the formula but fumbled the z-values from memory mid-call.
Start by clarifying the parameters and the formula for sample size calculation for two proportions. Then compute the baseline and treatment conversion rates, plug into the formula, and present the numeric result clearly. Mention any assumptions and discuss practical implications.
Pro tip: Always state the formula and assumptions explicitly, and round up the sample size to ensure sufficient power. Also, mention that this is a simplified calculation and real-world adjustments (e.g., for multiple comparisons, sequential testing) may be needed.
Confirm the baseline conversion rate (4%), relative uplift (10%), alpha (0.05 two-sided), and power (80%). State that the test is two-sided and assume equal variance for the sample size calculation.
Calculate the treatment conversion rate: 4% * (1 + 10%) = 4.4%. The absolute difference (delta) is 0.4 percentage points (0.004).
Use the standard formula for two proportions: n = (Z_{α/2} + Z_{β})^2 * (p1*(1-p1) + p2*(1-p2)) / (p2 - p1)^2. Plug in Z_{α/2}=1.96, Z_{β}=0.84, p1=0.04, p2=0.044.
Calculate the numerator and denominator: (1.96+0.84)^2 = 7.84; p1*(1-p1)=0.0384, p2*(1-p2)=0.042064, sum=0.080464; delta^2=0.000016. So n = 7.84 * 0.080464 / 0.000016 ≈ 39,427 per variant. Round up to 39,428.
State that approximately 39,428 users per variant are needed. Discuss that this is a large sample, and mention potential adjustments for real-world constraints (e.g., multiple metrics, sequential testing).
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Bonferroni just divides alpha by 2, so you use 0.025 instead of 0.05, which bumps the z-value to about 2.24 and inflates sample size by maybe 30%.
First, explain how to apply the Bonferroni correction to the sample size calculation for two primary metrics, then compare the trade-offs of Bonferroni against using a gatekeeper metric or hierarchical testing. Focus on statistical power, false positive control, and practical implications for decision-making.
Pro tip: Emphasize that Bonferroni is conservative and can inflate sample size requirements, so in practice, consider the cost of false positives versus false negatives and the business impact of each metric. Mention that hierarchical testing preserves power for the primary metric while still allowing secondary analyses.
Define conversion rate and qualified-lead rate as co-primary metrics. Explain that Bonferroni correction adjusts the significance level (α) by dividing it by the number of metrics (k=2), so α' = α/2.
Show that the required sample size per variant increases because the effective α is smaller. Use the formula n ∝ (z_{1-α'/2} + z_{1-β})^2 / Δ^2, and note that reducing α increases the z-value, thus requiring a larger n.
Discuss that Bonferroni controls the family-wise error rate (FWER) but is conservative, reducing power and increasing sample size. It may be impractical if metrics are correlated or if one metric is more important.
Describe gatekeeper approach: test one primary metric first; only if significant, test the second. This preserves α for the primary but may miss effects on the secondary if the primary fails. It prioritizes one metric over another.
Explain hierarchical (fixed-sequence) testing: order metrics by importance, test sequentially without alpha adjustment, stopping when a non-significant result occurs. This controls FWER and maintains power for the primary metric, but requires a priori ordering and may not test all metrics if early ones fail.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
The arithmetic is straightforward: 1.2M times 0.7 is 840K eligible, 20% of that is 168K per day across both arms, so 84K per variant per day.
Start by calculating the daily eligible traffic and the experiment traffic under the 20% cap. Then determine the required sample size per variant based on desired statistical power and minimum detectable effect, and divide by daily traffic to get the duration. Finally, propose a ramp schedule that gradually increases traffic while monitoring for guardrail metric regressions.
Pro tip: Always state your assumptions explicitly (e.g., baseline conversion rate, MDE, power) and show that you understand the trade-off between test duration and sensitivity. Mention that you would validate the ramp with a pre-period and use sequential testing or CUPED to maintain validity.
Compute 70% of 1.2M to get 840k eligible sessions per day. Then apply the 20% cap to get 168k sessions per day in the experiment.
Use power analysis: assume baseline conversion rate, minimum detectable effect, significance level (α=0.05), and power (1-β=0.8). Calculate total sample size needed (e.g., using formula for two proportions).
Divide the total required sample size by the daily experiment traffic (168k) to get the number of days. Round up to account for weekly seasonality (e.g., full weeks).
Start with a small percentage (e.g., 5% of eligible traffic) for a few days to monitor guardrail metrics, then increase gradually (e.g., 10%, 20%) over 1-2 weeks until reaching the cap, ensuring each ramp step has enough data to detect issues.
Use sequential testing or always-valid p-values to allow peeking, and consider CUPED to reduce variance. Maintain consistent assignment and avoid mixing ramp periods in the final analysis.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Z-test is fine here because sample sizes are large and Bernoulli outcomes let you invoke CLT without much worry.
Start by clarifying the assumptions and context of the experiment (sample size, known variance, normality). Then explain the decision rule between z-test and t-test, and finally interpret the borderline confidence interval with nuance about practical significance and uncertainty.
Pro tip: At Meta, with large samples, the z-test and t-test often yield nearly identical results; focus on the effect size and business impact rather than just statistical significance. A confidence interval barely excluding zero suggests a small effect that may not be practically meaningful—always consider the confidence interval width and the cost of a false positive.
Ask about sample size, whether population variance is known, and the distribution of the metric. This determines the appropriate test.
Use z-test when population variance is known and sample size is large (n>30) or data is normal. Use t-test when variance is unknown and estimated from the sample, especially with small samples.
In practice, with large samples (common in tech), t-test and z-test converge. However, t-test is more robust to unknown variance and is default in many tools.
A 95% CI barely excluding zero means the effect is statistically significant but likely small. Consider the lower bound: if it's close to zero, the effect could be negligible. Also, 5% of such intervals will exclude zero by chance even if true effect is zero.
Assess whether the effect size is meaningful for the product. A tiny but significant lift might not justify implementation costs. Also consider the confidence interval width and potential for practical significance.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Standard SRM check first, compare observed traffic split to expected 50/50 using a chi-square test.
Start by explaining that randomization checks are a pre-experiment validation step to ensure internal validity. Describe a systematic process: first verify the randomization mechanism, then compare key dimensions between groups using statistical tests, and finally investigate any imbalances. Emphasize that while some imbalance is expected by chance, systematic differences indicate a problem.
Pro tip: Use standardized mean differences (SMD) rather than p-values for balance checks, as p-values are sensitive to sample size and can flag trivial imbalances in large experiments. Also, pre-register the dimensions and thresholds for imbalance to avoid post-hoc rationalization.
Check that the randomization unit (e.g., user, session) was correctly assigned to treatment/control using the intended algorithm (e.g., hash-based). Ensure no bias in assignment, such as unequal probabilities or leakage.
List pre-experiment covariates that could affect the outcome, such as country, platform, user demographics, and historical behavior. These should be measured before the experiment starts.
For each dimension, compute summary statistics (e.g., mean, proportion) for treatment and control. Use standardized mean differences (SMD) or statistical tests (e.g., t-test, chi-square) to assess balance.
Check if any dimension exceeds a pre-defined threshold (e.g., SMD > 0.1). If imbalances are found, investigate potential causes (e.g., randomization bug, sample ratio mismatch) and consider whether they bias results.
If randomization is flawed, halt the experiment or apply statistical adjustments (e.g., CUPED, stratification). If balanced, proceed with analysis, noting any minor imbalances as limitations.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.