Start by clarifying the data structure and the null hypothesis (no difference between each treatment and control). Then, for each variant, compute the z-statistic for the difference in proportions and derive the two-sided p-value using the normal approximation. Finally, discuss multiple testing correction and practical significance.
Pro tip: Always apply a multiple comparisons correction (e.g., Bonferroni or Benjamini-Hochberg) when testing multiple variants against a single control, and report both raw and adjusted p-values to avoid false positives.
Confirm the conversion counts and sample sizes for control and each treatment. State the null hypothesis: the conversion rate for each treatment equals that of the control.
Calculate the conversion rate for each group (conversions / 50,000). Compute the pooled proportion and standard error for each treatment-control comparison.
For each variant, compute the z-statistic as (p_treatment - p_control) / SE, then find the two-sided p-value using the normal CDF.
Apply a correction method (e.g., Bonferroni or Benjamini-Hochberg) to control the family-wise error rate or false discovery rate across the two comparisons.
Report p-values, confidence intervals, and effect sizes. Consider whether the observed differences are practically meaningful, not just statistically significant.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Bonferroni is the easy answer and I went with it, adjusted alpha to 0.025 per test.
Start by explaining the multiple comparisons problem in the context of A/B testing with two variants, then propose a correction method like Bonferroni or Benjamini-Hochberg, and finally demonstrate how to apply it to adjust p-values and draw conclusions. Emphasize the trade-off between controlling false positives and maintaining statistical power.
Pro tip: Mention that the choice of correction depends on the goal: Bonferroni is conservative and best when false positives are costly, while FDR methods like Benjamini-Hochberg are more powerful when exploring multiple metrics. Also, consider using sequential testing or Bayesian methods as alternatives.
Explain that testing two variants simultaneously increases the chance of false positives due to multiple hypotheses (e.g., each variant vs. control). Quantify the inflated Type I error rate if uncorrected.
Select a method based on the context: Bonferroni for strict family-wise error control, or Benjamini-Hochberg for false discovery rate control when multiple metrics are tested. Justify your choice.
Show how to adjust the p-values: for Bonferroni, multiply each p-value by the number of comparisons (or divide alpha); for Benjamini-Hochberg, rank p-values and compare to (i/m)*Q. Provide a concrete example.
Compare adjusted p-values to the significance level and state which variants are statistically significant. Discuss the impact on conclusions, including potential loss of power and the need for larger sample sizes.
Acknowledge the trade-off between Type I and Type II errors. Mention alternatives like sequential testing, Bayesian methods, or pre-registering a primary metric to avoid the issue.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Blanked for a second on the relative uplift CI formula.
Start by clarifying the metric and data structure, then compute the absolute uplift (difference in means) and relative uplift (ratio of means) for each variant vs control. Use the delta method or bootstrap to derive standard errors and construct 95% confidence intervals, ensuring to account for multiple comparisons if needed.
Pro tip: Always check the assumptions behind the delta method (e.g., large sample, non-zero control mean) and consider using bootstrap for robustness, especially with skewed metrics. Also, present both absolute and relative intervals because stakeholders often care about different aspects.
Confirm the metric (e.g., conversion rate, revenue per user) and the experimental design (sample sizes, randomization). Ensure you have the necessary summary statistics (means, variances, sample sizes) for each group.
Calculate the difference in means between variant and control. For the CI, use the standard error of the difference: sqrt(var_variant/n_variant + var_control/n_control), then construct a 95% CI using the normal approximation (or t-distribution if small samples).
Calculate the ratio of means (variant/control). Use the delta method to approximate the variance of the ratio: (mean_variant^2 / mean_control^2) * (var_variant/(n_variant*mean_variant^2) + var_control/(n_control*mean_control^2)). Alternatively, use bootstrap resampling to empirically derive the CI.
If comparing multiple variants to a single control, consider using a correction like Bonferroni or Dunnett's test to control the family-wise error rate. This affects the confidence level or the critical value used.
Present the CIs for both absolute and relative uplift, noting whether they include zero (for absolute) or one (for relative). Discuss practical significance and any caveats (e.g., assumptions, sample size).
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
This is where the conversation got interesting.
Start by clearly stating your ship/no-ship recommendation based on the primary metric and guardrails, then outline the additional checks you'd run to validate the result (e.g., novelty effects, segment analysis, SRM). Finally, explain how you'd communicate uncertainty to non-technical stakeholders using plain language, confidence intervals, and business impact framing.
Pro tip: Always tie the recommendation to business impact and risk tolerance—stakeholders care more about potential revenue or user experience changes than p-values. Use a 'confidence score' or 'range of outcomes' to make uncertainty tangible.
Give a clear ship or no-ship call based on the primary metric and guardrail metrics. If results are ambiguous, recommend a conditional ship (e.g., ship to a subset) or extend the test.
Run additional checks: sample ratio mismatch (SRM), novelty/primacy effects, segment-level consistency, and sensitivity analyses. Ensure the result is robust and not driven by outliers or a single segment.
Calculate confidence intervals and effect sizes. Consider Bayesian methods for a more intuitive probability of superiority. Assess practical significance vs. statistical significance.
Translate statistical uncertainty into business terms: e.g., 'We are 85% confident the change will increase conversion by 1-3%, which could mean $X in annual revenue.' Use visualizations like confidence intervals or probability distributions.
Suggest a phased rollout with monitoring, or additional tests to reduce uncertainty. Define success metrics and guardrails for the rollout phase.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.