The derivation itself isn't bad if you've done it before.
Start by clearly stating the normal approximation formula for per-arm sample size in a two-proportion z-test, then plug in the baseline conversion rate, absolute MDE, and standard z-scores for the desired significance level and power. Compute the result step-by-step, showing all arithmetic and the final sample size per arm.
Pro tip: Always clarify the assumptions: two-sided test at 5% significance and 80% power, and mention that this is a simplified calculation that ignores practical considerations like novelty effects or multiple testing. Also, note that the formula assumes equal variance under null and alternative, which is a common approximation.
Write the standard sample size formula for comparing two proportions: n = ( (z_{α/2} + z_β)^2 * (p1(1-p1) + p2(1-p2)) ) / (p2 - p1)^2, where p1 is baseline and p2 is baseline + absolute MDE.
Identify p1 = 0.08, relative MDE = 8%, so absolute MDE = 0.08 * 0.08 = 0.0064, and p2 = 0.0864. Set α = 0.05 (two-sided) so z_{α/2} = 1.96, and power = 0.80 so z_β = 0.84.
Calculate p1(1-p1) = 0.08 * 0.92 = 0.0736 and p2(1-p2) = 0.0864 * 0.9136 ≈ 0.07894. Sum them to get 0.15254.
Compute (1.96 + 0.84)^2 = 2.8^2 = 7.84. Multiply by sum of variances: 7.84 * 0.15254 ≈ 1.1959. Divide by (0.0064)^2 = 0.00004096 to get n ≈ 29192 per arm.
Round up to the nearest whole number: approximately 29,192 per arm. Mention that this is a large sample, typical for detecting small relative effects.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Six sub-scenarios in one question is a lot.
First, establish the baseline sample size formula for a two-sample proportion test and then systematically apply each change, explaining the qualitative direction and quantitative multiplier. For clustering and sequential monitoring, adjust the effective sample size using the design effect and the alpha-spending function, respectively.
Pro tip: Emphasize that these changes compound multiplicatively, so the total sample size increase can be dramatic—quantify the combined effect to show you understand the practical implications for experiment duration and traffic allocation.
Start with the standard formula for two-sample proportion test: n = (Z_{1-α/2} + Z_{1-β})^2 * (p1(1-p1) + p2(1-p2)) / (p1 - p2)^2, assuming equal allocation. This provides the reference point for all adjustments.
Halving MDE quadruples sample size (since n ∝ 1/MDE^2). Tightening alpha to 0.01 increases Z_{1-α/2} from 1.96 to 2.576, and increasing power to 90% increases Z_{1-β} from 0.84 to 1.28. Compute the combined multiplier for these three changes.
For 75/25 allocation, the sample size increases by a factor of 1/(4 * 0.75 * 0.25) = 1/0.75 ≈ 1.333 compared to equal allocation, assuming the same total sample size. Alternatively, use the formula with allocation ratio k = 0.25/0.75 = 1/3, which gives a multiplier of (1+k)^2 / (4k) = (4/3)^2 / (4/3) = 4/3 ≈ 1.333.
With ICC=0.02 and average cluster size m=5, the design effect (DEFF) = 1 + (m-1)*ICC = 1 + 4*0.02 = 1.08. Multiply the sample size by DEFF to account for user-level clustering.
Using two equally spaced looks with O'Brien-Fleming boundaries, the maximum sample size increases by approximately 5-10% (e.g., 1.05-1.10 multiplier) to maintain power. The exact inflation factor depends on the number of looks and alpha-spending function; for two looks, it's about 1.05.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Start by explaining that standard sample size formulas assume normally distributed data, so for overdispersed negative binomial counts you need to adjust for the variance-mean relationship. Then walk through the process: estimate the dispersion parameter from historical data, derive the variance under the negative binomial, and plug it into the sample size formula for count data (e.g., using the coefficient of variation or the negative binomial test).
Pro tip: Mention that you would validate the dispersion estimate using a holdout set or pilot data, and consider using simulation-based power analysis to account for uncertainty in the dispersion parameter—this shows practical maturity beyond textbook formulas.
Confirm the metric is a count and check for overdispersion by comparing variance to mean. If variance >> mean, negative binomial is appropriate.
Use historical data or a pilot to estimate the dispersion parameter (often denoted as alpha or k) via maximum likelihood or method of moments.
For negative binomial, variance = mean + mean^2 / theta (or mean + alpha*mean^2). Use this to compute the standard error for the test statistic.
Use the negative binomial variance in the sample size formula for comparing two means (e.g., n = 2*(z_alpha/2 + z_beta)^2 * variance / effect_size^2). Alternatively, use simulation-based power analysis.
Check the calculated sample size via simulation or pilot data, and refine if the dispersion estimate is uncertain. Consider sensitivity analysis for different dispersion values.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.