I knew the formula but fumbled the variance calculation at first.
Start by clearly stating the assumptions and the formula for sample size per arm for a two-proportion z-test. Then plug in the baseline rate (9% = 0.09) and the target rate (9.5% = 0.095), compute the required sample size per arm, and finally divide by daily eligible users (10 million) to get the number of days, accounting for equal allocation.
Pro tip: Always sanity-check the sample size with a quick approximation: for small lifts, the required sample per arm is roughly 16 * p(1-p) / (delta^2). This helps catch calculation errors and shows you understand the underlying statistics.
State the baseline conversion rate (p1 = 0.09), target rate (p2 = 0.095), absolute lift (delta = 0.005), significance level (alpha = 0.05, two-sided), power (1 - beta = 0.80), and equal allocation ratio.
Use the standard formula for two-proportion z-test: n per arm = ( (z_{1-alpha/2} * sqrt(2 * p_bar * (1-p_bar)) + z_{1-beta} * sqrt(p1*(1-p1) + p2*(1-p2)) )^2 ) / (delta^2), where p_bar = (p1+p2)/2.
Plug in z_{0.975} = 1.96, z_{0.80} = 0.84, p1 = 0.09, p2 = 0.095, delta = 0.005. Calculate p_bar = 0.0925. Then compute the required n per arm. (Result: approximately 1,048,000 per arm, but show the calculation.)
Total sample needed = 2 * n per arm. With 10 million eligible users per day and equal allocation, daily users per arm = 5 million. Days = n per arm / 5,000,000. Round up to the nearest whole day.
Mention that the calculated duration assumes all eligible users are exposed and that the metric is stable. Discuss potential adjustments for novelty effects, seasonality, or multiple testing, and the importance of checking for sample ratio mismatch (SRM).
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Sample size scales with variance, so a 20% higher SD means variance goes up by 44% (1.2 squared).
Start by recalling the sample size formula for comparing means, which is proportional to the variance (σ²). Then explain that a 20% increase in standard deviation leads to a 44% increase in required sample size (since 1.2² = 1.44). Finally, discuss practical implications such as the need to adjust the experiment duration or accept reduced power.
Pro tip: Mention that in practice, you can mitigate the impact by using a more sensitive metric (e.g., relative instead of absolute) or by employing sequential testing to allow early stopping, but always pre-register the analysis plan to avoid p-hacking.
State the standard formula for sample size per group in a two-sample t-test: n = 2*(Zα/2 + Zβ)² * σ² / Δ², where σ is the standard deviation and Δ is the minimum detectable effect.
Note that sample size is directly proportional to the variance (σ²). Therefore, a 20% increase in σ results in a 44% increase in required sample size (1.2² = 1.44).
If the original sample size was n, the new required sample size becomes 1.44n. For example, if n was 10,000 per group, it would increase to 14,400 per group.
Explain that this increase means longer experiment duration, higher cost, or reduced power if the sample size is fixed. Suggest options like increasing the sample size, accepting lower power, or using a more sensitive metric.
Advise using pilot data or historical data to better estimate σ, and consider variance reduction techniques (e.g., CUPED) to mitigate the impact of higher-than-expected variance.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Start by explaining that with two treatment arms compared to a shared control, you have two primary comparisons, so you need to control the familywise error rate (FWER) at 5%. Recommend a multiple comparison correction such as Bonferroni or Holm-Bonferroni, and justify why it's appropriate for this small number of comparisons. Then, recalculate the per-arm sample size using the corrected alpha, and adjust for the unequal traffic allocation (20% control, 40% each treatment) to determine the total sample size and duration.
Pro tip: Mention that while Bonferroni is simple and controls FWER, it can be conservative; if the comparisons are independent, you might consider a slightly less conservative approach like Holm-Bonferroni, but for two comparisons the difference is minimal. Also, emphasize that the shared control means the control data is used for both comparisons, so the effective sample size for control is larger, which can be leveraged in power calculations.
Recognize that there are two primary comparisons (Treatment A vs Control, Treatment B vs Control). To maintain a familywise error rate of 5%, you need to adjust the per-comparison alpha.
Select Bonferroni or Holm-Bonferroni. For two comparisons, Bonferroni splits alpha into 0.025 per comparison. Explain the trade-off between FWER control and power.
Use the standard sample size formula for two proportions with alpha = 0.025 (two-sided) and power = 80% (or 90%). Compute the required sample size per arm assuming equal allocation first.
Given 20% to control and 40% to each treatment, the allocation ratio is 1:2:2. Recalculate the total sample size needed to achieve the same power, accounting for the unequal split. The control group will need to be larger relative to each treatment to maintain power.
Using the total sample size and the daily traffic rate (if provided, otherwise assume a rate), calculate the number of days required to complete the experiment.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Start by explaining the multiple comparisons problem and how daily peeking inflates the Type I error rate. Then describe how sequential testing methods like O'Brien-Fleming adjust boundaries to control the overall alpha while allowing early stopping. Finally, discuss the trade-offs and practical implications for experiment design and decision-making.
Pro tip: Mention that O'Brien-Fleming is conservative early on, which is ideal when you want to avoid stopping too early on noise, but it requires pre-planning the number and timing of looks. Also, note that many tech companies use sequential testing to enable continuous monitoring without inflating false positives.
Describe how repeatedly testing at p < 0.05 each day increases the chance of a false positive (Type I error) due to multiple comparisons. Quantify the inflation: e.g., with 20 peeks, the false positive rate can exceed 30%.
Define sequential testing as a method that adjusts significance thresholds at each interim analysis to control the overall alpha. Explain that O'Brien-Fleming uses conservative boundaries early (very small alpha) and more lenient boundaries later, minimizing the risk of early stopping on noise.
Detail that O'Brien-Fleming boundaries are based on the cumulative alpha spent, often using a function like the Lan-DeMets approach. Early looks require extremely small p-values (e.g., p < 0.0001), while the final look uses a threshold close to the nominal alpha (e.g., p < 0.05).
Explain that sequential testing requires pre-specifying the number and timing of interim analyses. It allows for early stopping if effects are strong, but may delay decisions if effects are moderate. Also, note that it controls false positives but can increase sample size if the effect is small.
Summarize that naive daily checks are invalid, and sequential testing like O'Brien-Fleming provides a rigorous framework for monitoring experiments. Recommend using it when continuous monitoring is desired, and always pre-register the analysis plan.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
I talked about intraclass correlation and how if users within a cluster behave similarly, your effective sample size is smaller than the raw count.
Explain that the choice between standard variance reduction and cluster-robust standard errors depends on the correlation structure of the data and the unit of randomization. Discuss how clustering at a higher level (e.g., feed) accounts for within-cluster correlation, and describe the impact on sample size requirements, including the design effect and effective sample size.
Pro tip: Always consider the trade-off between bias and variance: cluster-robust standard errors are consistent but less efficient, so you need a larger sample to achieve the same power. Also, remember that the clustering level should match the level at which treatment is assigned or where interference occurs.
Determine if there is correlation among observations within clusters (e.g., users within a feed, or multiple observations per user). This often arises from group randomization or interference.
Explain that ignoring clustering leads to underestimated standard errors and inflated false positive rates. Cluster-robust standard errors adjust for this by allowing arbitrary correlation within clusters.
Switch when the intra-cluster correlation (ICC) is non-negligible, or when the treatment is assigned at a cluster level. Also consider if there is interference between units within a cluster.
Introduce the design effect (DEFF) = 1 + (m-1)*ICC, where m is the average cluster size. The required sample size increases by this factor to maintain power.
Discuss how larger sample sizes may be infeasible, and potential alternatives like using more clusters, reducing cluster size, or using more efficient estimators (e.g., fixed effects).
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.