This is the foundation question and I spent too long second-guessing whether the 7% was relative or absolute.
Start by converting the relative MDE to an absolute effect size, then apply the standard two-proportion sample size formula for a two-sided test. Clearly state each formula and plug in the values, showing intermediate calculations to demonstrate transparency.
Pro tip: Always sanity-check your result against a known rule of thumb (e.g., ~16 * p(1-p)/delta^2 for 80% power and 5% alpha) and mention that real-world factors like traffic allocation or multiple testing may require adjustments.
Identify baseline conversion rate (p1 = 0.18), relative MDE (7%), and compute absolute MDE (delta = 0.18 * 0.07 = 0.0126). Set p2 = p1 + delta = 0.1926.
Use n = (Z_{α/2} + Z_β)^2 * (p1(1-p1) + p2(1-p2)) / (p2 - p1)^2 for equal allocation. For α=0.05 two-sided, Z_{α/2}=1.96; for 80% power, Z_β=0.84.
Calculate p1(1-p1)=0.18*0.82=0.1476, p2(1-p2)=0.1926*0.8074≈0.1555, sum=0.3031. (1.96+0.84)^2=7.84. Delta^2=0.0126^2=0.00015876. n = 7.84 * 0.3031 / 0.00015876 ≈ 14968 per arm.
Round up to the nearest whole number (e.g., 14,968 per arm). Explain that this is the required sample size per variant to detect the specified effect with 80% power and 5% significance.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
CUPED variance reduction is just 1 minus R-squared, so 25% variance reduction here.
Start by explaining the CUPED variance reduction formula: variance reduction equals R-squared of the pre-experiment covariate with the outcome. Then apply it to the given R-squared of 0.25 to conclude a 25% reduction in variance, which translates to a 25% reduction in required sample size (since sample size is proportional to variance). Finally, discuss practical implications and assumptions.
Pro tip: Mention that the R-squared must be estimated from historical data and that CUPED is most effective when the covariate is highly correlated with the outcome; also note that the reduction in sample size is approximate and assumes the covariate is properly used.
State that CUPED reduces variance by a factor of (1 - R^2), where R^2 is the squared correlation between the pre-experiment covariate and the outcome metric.
Plug in R^2 = 0.25 to get variance reduction = 1 - 0.25 = 0.75, meaning variance is reduced by 25%.
Since required sample size is proportional to variance, a 25% reduction in variance leads to a 25% reduction in required sample size (e.g., from N to 0.75N).
Note that this assumes the covariate is pre-experiment, properly implemented, and that the R-squared estimate is accurate; also mention that CUPED can be combined with other variance reduction techniques.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Design effect comes out to 1 + (0.8)(0.06) = 1.048.
First, compute the design effect (DE) using the given formula with m=1.8 and ICC=0.06, yielding DE = 1 + (1.8-1)*0.06 = 1.048. Then, multiply the standard per-arm sample size (without clustering) by DE to get the adjusted per-arm size for household-level randomization. For CUPED, apply the variance reduction factor (1 - ρ²) to the adjusted sample size, where ρ is the correlation between the pre-experiment covariate and the outcome; if ρ is not provided, state the formula and note that the CUPED-adjusted size is the household-adjusted size times (1 - ρ²).
Pro tip: Always clarify that the design effect inflates the sample size to account for intra-cluster correlation, and that CUPED further reduces it by leveraging pre-experiment data; if the baseline sample size isn't given, express the answer as a multiplier (e.g., 'multiply the standard per-arm size by 1.048 for household clustering, and by 1.048*(1-ρ²) with CUPED').
Note the average household size (m=1.8), ICC=0.06, and that the baseline per-arm sample size (n) is not provided. State that you will express the adjusted sizes as multipliers of n.
Apply the formula DE = 1 + (m-1)*ICC = 1 + (1.8-1)*0.06 = 1.048. Explain that this factor accounts for the correlation among individuals within the same household.
Multiply the baseline per-arm size by DE: n_adj = n * 1.048. If n is known, provide the numeric value; otherwise, state the multiplier.
Apply the CUPED variance reduction factor (1 - ρ²), where ρ is the correlation between the pre-experiment covariate and the outcome. The CUPED-adjusted per-arm size is n_adj * (1 - ρ²). If ρ is not given, present the formula and note that typically ρ ranges from 0.3 to 0.7, leading to further sample size reduction.
Conclude that household-level randomization increases the required sample size by 4.8%, while CUPED can reduce it by a factor of (1 - ρ²). Emphasize that both adjustments are essential for accurate power analysis in clustered randomized experiments.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
The key point is that O'Brien-Fleming is conservative early and spends very little alpha at the first looks, which protects you from stopping too soon on noise.
Start by defining O'Brien-Fleming boundaries as a group sequential design that spends very little alpha early and more later, preserving overall Type I error. Then explain the practical implications: you need larger effect sizes to stop early, but you gain flexibility to monitor without inflating false positives, which affects timeline and power calculations.
Pro tip: Mention that O'Brien-Fleming is conservative early, so you rarely stop for efficacy at the first interim, but it's excellent for safety monitoring. Also, highlight that you must pre-specify the number and timing of interim looks to control alpha properly.
Explain that O'Brien-Fleming is a group sequential method where the alpha spent at each interim look is very small early and increases toward the final analysis, following a Lan-DeMets spending function.
Describe how the overall Type I error rate (e.g., 5%) is partitioned across interim looks, with minimal alpha spent at early looks and most reserved for the final look, thus controlling the family-wise error rate.
Note that early stopping requires a very large treatment effect (stringent boundary), making it hard to stop early for efficacy, but easy to stop for harm. Later looks have less stringent boundaries.
Explain that because early stopping is unlikely, the experiment often runs to the planned end, so timeline is more predictable. Power is slightly reduced compared to a fixed-sample design if you don't stop early, but the loss is minimal with few interim looks.
Suggest using O'Brien-Fleming when early stopping for efficacy is not critical, but safety or futility monitoring is important. Pre-specify interim looks and adjust sample size to maintain desired power.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Benjamini-Hochberg: rank p-values, compare each to (rank/total) times FDR threshold, reject up to the largest one that passes.
Start by explaining that with 12 secondary metrics, you need to control the false discovery rate (FDR) to avoid false positives. Describe using a method like Benjamini-Hochberg to control FDR at 10%, and then discuss how to communicate that a 'discovery' means a metric is statistically significant after FDR control, but still requires practical significance and business context.
Pro tip: Emphasize that FDR control is about managing the expected proportion of false discoveries among all discoveries, not the probability of any false positive. This shows a nuanced understanding that impresses interviewers.
Acknowledge that testing 12 metrics inflates the chance of false positives. Explain that without correction, the family-wise error rate would be too high.
Propose using the Benjamini-Hochberg procedure to control FDR at 10%. Explain that it ranks p-values and compares them to thresholds based on rank and total tests.
Detail that you would apply BH to the 12 secondary metrics, possibly after confirming primary metric significance. Note that FDR control allows some false discoveries but limits their proportion.
Explain that a 'discovery' means the metric's p-value passed the BH threshold, so it's statistically significant after FDR control. But stress that it's not definitive proof; consider effect size, confidence intervals, and business relevance.
Describe how to convey results to non-technical stakeholders: use plain language, avoid jargon, and emphasize that discoveries are hypotheses for further validation, not final truths.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
ITT uses everyone as assigned regardless of what they saw, CACE adjusts for the actual compliance rate.
First, clarify that ITT measures the effect of assignment to treatment, while CACE measures the effect among compliers. Then, compute both using the given noncompliance and contamination rates, and discuss how to present them to a product team by emphasizing the trade-off between real-world impact and efficacy.
Pro tip: Always present ITT as the conservative, policy-relevant estimate and CACE as the per-protocol effect, but caution that CACE relies on assumptions like exclusion restriction and monotonicity. Use a simple analogy, such as comparing the effect of offering a coupon (ITT) versus the effect on those who actually use it (CACE).
Explain that ITT estimates the effect of treatment assignment, regardless of compliance, while CACE estimates the effect among those who comply with their assigned treatment.
Given 10% noncompliance in treatment and 3% contamination in control, calculate the ITT effect as the difference in outcomes between the treatment and control groups as assigned.
Use the formula CACE = ITT / (compliance rate in treatment - contamination rate in control) to estimate the effect among compliers.
Present ITT as the expected impact if the treatment is rolled out to the entire population, and CACE as the potential impact if perfect compliance were achieved. Highlight that CACE is an estimate with assumptions and may not reflect real-world scalability.
Mention key assumptions for CACE (e.g., exclusion restriction, monotonicity) and caution that violations could bias the estimate. Recommend using ITT for decision-making unless there's a strong reason to focus on compliers.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.