The arithmetic itself is fine but I fumbled the CI for relative lift.
Start by clearly defining the metrics: absolute lift is the difference in CTRs, relative lift is the ratio of that difference to the control CTR. Then compute the pooled standard error using the pooled proportion, calculate the z-statistic, find the two-sided p-value, and construct a 95% confidence interval for the relative lift using the delta method or normal approximation. Finally, interpret the results in the context of statistical and practical significance.
Pro tip: Always state your assumptions (e.g., independence, large sample) and mention that with such large samples, even tiny lifts can be statistically significant but may not be practically meaningful. Also, consider using the delta method for the confidence interval of the relative lift to account for the ratio's non-linearity.
Compute the absolute lift as the difference between treatment and control CTRs. Then compute the relative lift as the absolute lift divided by the control CTR, often expressed as a percentage.
Calculate the pooled CTR by combining successes from both groups. Then compute the pooled standard error using the formula sqrt(p_pool*(1-p_pool)*(1/n_control + 1/n_treatment)).
Compute the z-statistic as the absolute lift divided by the pooled standard error. Then find the two-sided p-value using the standard normal distribution.
Use the delta method or normal approximation to compute the standard error of the relative lift, then construct the confidence interval as relative_lift ± 1.96 * SE_relative.
Discuss statistical significance (p-value < 0.05) and practical significance (effect size). Mention any caveats such as multiple testing or novelty effects.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Knew peeking was bad, knew it inflated Type I error, but quantifying it precisely under 14 peeks was where I got vague.
Start by explaining that peeking without correction inflates Type I error, then quantify the inflation using the multiple testing framework (e.g., 14 peeks at α=0.05 can inflate to ~20-30%). Finally, describe two solutions: sequential testing with alpha spending (e.g., O'Brien-Fleming) and Bayesian approach with a prior on lift, highlighting trade-offs.
Pro tip: Mention that the inflation depends on the correlation between interim looks and the number of peeks; in practice, daily peeking over 14 days can inflate Type I error to around 20-30%, but exact value requires simulation. Also, note that Bayesian methods with a skeptical prior can control false positives while providing interpretable posterior probabilities.
Explain that each peek is a hypothesis test, so with 14 independent tests at α=0.05, the family-wise error rate is 1 - (0.95)^14 ≈ 51%. However, since tests are correlated, the actual inflation is lower, typically around 20-30% for daily peeking. Suggest simulation to get exact value.
Highlight that without correction, the false positive rate increases, leading to more false discoveries and undermining trust in experiments. This is especially critical in tech companies where many experiments run concurrently.
Describe alpha spending functions (e.g., O'Brien-Fleming, Pocock) that allocate alpha across interim analyses. Explain that the overall Type I error is controlled at the desired level (e.g., 5%) by adjusting thresholds at each peek. Mention that this requires pre-specifying the number and timing of looks.
Explain that a Bayesian approach with an informative prior (e.g., skeptical prior centered at zero) can regularize the lift estimate and control false positives. The posterior probability of superiority can be monitored continuously without inflating error rates, as long as the prior is pre-registered.
Discuss trade-offs: sequential testing is frequentist and requires planning, while Bayesian is more flexible but sensitive to prior choice. Recommend based on context, e.g., for Meta's scale, sequential testing is common, but Bayesian is gaining traction for its interpretability.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Mentioned Bonferroni for FWER and Benjamini-Hochberg for FDR.
Start by acknowledging the multiple comparisons problem and its impact on false positives. Then, discuss methods to control familywise error rate (e.g., Bonferroni, Holm-Bonferroni) and false discovery rate (e.g., Benjamini-Hochberg), and explain how to choose based on the goal (strict control vs. discovery). Finally, emphasize practical considerations like pre-registration, hierarchical testing, and the trade-off between power and error control.
Pro tip: At Meta, guardrail metrics are often used to detect harm, so a conservative approach like Bonferroni or Holm is preferred to minimize false negatives, but you can also mention that if the goal is exploratory, FDR methods like Benjamini-Hochberg are more powerful. Also, highlight that guardrails are typically one-sided tests for harm, which affects adjustment.
Explain that testing 8 guardrails plus the primary metric inflates the chance of false positives, and that without correction, the familywise error rate (FWER) can exceed the nominal alpha.
Decide whether to control FWER (e.g., Bonferroni, Holm-Bonferroni) or FDR (e.g., Benjamini-Hochberg). FWER is stricter and suitable when any false positive is costly; FDR is more powerful when some false discoveries are acceptable.
For Bonferroni, divide alpha by the number of tests (e.g., 0.05/9 ≈ 0.0056). For Holm, order p-values and compare sequentially. For Benjamini-Hochberg, rank p-values and compare to (i/m)*alpha.
Mention that guardrails are often one-sided (testing for harm), so adjust alpha accordingly. Also, pre-register the testing plan to avoid p-hacking, and consider hierarchical testing where primary metric is tested first.
Acknowledge that corrections reduce power, especially with many metrics. Suggest alternatives like combining guardrails into a composite or using Bayesian methods to mitigate the issue.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.