Three questions bundled into one, which I didn't fully appreciate until I was halfway through the frequentist part and they asked about sample size.
Start by defining lift as the relative difference in conversion rates between test and control, then test significance using a two-proportion z-test or chi-square test. For sample size, use the standard formula for two proportions with 90% power and 5% significance, and for Bayesian framing, shift to posterior distributions and probability of superiority.
Pro tip: Always clarify the baseline conversion rate and whether the 5% lift is relative or absolute, as this drastically changes sample size. Also, mention that Bayesian methods can provide more intuitive results for stakeholders but require prior specification.
Clarify conversion rate, lift definition (relative vs absolute), and set up null and alternative hypotheses for frequentist testing.
Compute conversion rates for test and control, calculate lift, and perform a two-proportion z-test to get a p-value and confidence interval.
Use the formula for two proportions: n = (Zα/2 + Zβ)^2 * (p1(1-p1) + p2(1-p2)) / (p2-p1)^2, with p2 = p1 * 1.05 for a 5% relative lift.
Model conversion rates with Beta priors, update with data to get posteriors, and compute the probability that test > control or the expected lift distribution.
Explain frequentist and Bayesian results in business terms, highlighting uncertainty and decision-making implications.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.