The design effect formula itself isn't hard once you remember DE = 1 + (m-1)*ICC, but I fumbled the algebra when converting the inflated variance back to cluster counts.
Start by deriving the sample size formula for individual randomization using the baseline rate and minimum detectable effect, then adjust for cluster randomization by multiplying by the design effect (1 + (m-1)*ICC). Finally, compute the number of clusters per arm by dividing the total required users by the cluster size, ensuring it's rounded up.
Pro tip: Always clarify whether the ICC is for the outcome or the effect, and consider using the coefficient of variation to account for varying cluster sizes, as Meta often deals with unequal cluster sizes in practice.
Use the standard formula for comparing two proportions: n = (Z_{α/2} + Z_{β})^2 * (p1(1-p1) + p2(1-p2)) / (p2 - p1)^2, where p1 is baseline rate and p2 = p1 + MDE.
Compute the design effect (DE) = 1 + (m - 1) * ICC, where m is the average cluster size and ICC is the intra-cluster correlation.
Multiply the individual sample size by the design effect to get the required number of users per arm: N_users = n * DE.
Divide the required users per arm by the cluster size m and round up to get the number of clusters per arm: k = ceil(N_users / m).
Check assumptions (e.g., normality, known ICC) and consider adjustments for unequal cluster sizes or finite population correction if needed.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
I went with a cluster-robust z-test on cluster means, which felt right, but I got a bit tangled explaining when to swap to something nonparametric.
Start by clarifying the clustering structure and the primary metric's distribution, then recommend a cluster-robust test like the cluster-robust t-test or a mixed-effects model. Discuss when nonparametric alternatives like the cluster bootstrap or permutation tests are preferable, emphasizing trade-offs between assumptions and power.
Pro tip: Mention that at Meta, cluster-robust standard errors are often the default for clustered experiments, but always check the number of clusters—if it's small, use a nonparametric method like the cluster bootstrap to avoid inflated Type I error.
Identify the clustering unit (e.g., user, session) and the primary metric's type (continuous, binary, count). Confirm whether clusters are balanced and the number of clusters.
For large number of clusters, use a cluster-robust t-test (e.g., with CR2 standard errors) or a mixed-effects model if you need to model random effects. For binary metrics, consider a cluster-robust logistic regression or GEE.
When the number of clusters is small (<50), the metric is skewed, or there are outliers, prefer nonparametric methods like the cluster bootstrap or permutation tests (randomization at cluster level).
Check assumptions of parametric tests (normality of cluster-level means, homoscedasticity). If violated, nonparametric methods may be more reliable, but they can have lower power.
State your preferred test based on the context, and explain the trade-offs (e.g., robustness vs. power, computational cost).
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Pretty standard but the 'jointly with practical significance' part is where people slip up.
Start by clarifying the experiment design, primary metric, and whether the metric is a mean or proportion, then specify the confidence level (typically 95%) and how you'd compute the interval (e.g., using the delta method or bootstrap for ratio metrics). Explain that you'd interpret the interval by checking whether it excludes zero and whether the entire interval lies above or below the pre-defined practical significance threshold (e.g., minimum detectable effect or business-relevant lift).
Pro tip: Meta often cares about practical significance over statistical significance, so emphasize that a statistically significant result with a confidence interval that barely crosses zero may not be worth shipping. Also, mention that you'd consider the confidence interval's width to assess precision and potentially run the experiment longer if the interval is too wide to make a decision.
Confirm the experiment's primary metric (e.g., click-through rate, revenue per user), the unit of analysis (user-level), and the confidence level (usually 95%).
For simple metrics like proportions, use the standard normal approximation; for ratio metrics or complex designs, use the delta method or bootstrap to account for variance.
Calculate the confidence interval for the difference between treatment and control (absolute or relative), and report it with the point estimate and confidence level.
Compare the interval to the pre-defined practical significance threshold (e.g., a 1% lift). If the entire interval is above the threshold, the effect is practically significant; if it includes zero, the result is inconclusive.
Based on the interval and thresholds, recommend whether to ship, iterate, or abandon the change, considering business impact and risk.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
I knew O'Brien-Fleming style spending was the answer they wanted.
Start by clarifying the goal: control the family-wise error rate (FWER) across four weekly looks while maintaining power. Then choose a sequential testing method (e.g., Pocock, O'Brien-Fleming, or alpha spending) and compute adjusted alpha at each look, explaining trade-offs. Finally, discuss practical considerations like peeking bias, stopping rules, and communication with stakeholders.
Pro tip: Mention that in practice, you'd pre-register the sequential plan and use a tool like gsDesign or sequential package to compute boundaries, and emphasize that the adjusted alpha depends on the chosen spending function and the correlation between test statistics.
Confirm the primary metric, hypothesis (two-sided vs one-sided), desired overall alpha (e.g., 0.05), power (e.g., 80%), and expected effect size. Discuss whether early stopping is for efficacy, futility, or both.
Select a method such as Pocock (constant alpha at each look), O'Brien-Fleming (conservative early, liberal late), or a flexible alpha spending function (e.g., Lan-DeMets). Explain the trade-offs in terms of Type I error control and power.
For four equally spaced looks, calculate the adjusted alpha levels. For example, with Pocock and overall alpha=0.05, each look uses alpha≈0.0182; with O'Brien-Fleming, the first look might use alpha≈0.005, increasing to ≈0.048 at the final look. Show the calculations or reference software.
Discuss how to handle unplanned peeks, missing data, and multiple metrics. Mention that the sequential plan should be pre-registered and that any deviation requires correction.
Conclude with a clear recommendation, emphasizing that the adjusted alpha preserves the overall Type I error rate and that the choice of method depends on the risk tolerance for early stopping.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Went hierarchical: primary first, guardrails only evaluated conditionally.
Start by distinguishing the primary metric from guardrails, then explain how to control family-wise error rate (FWER) or false discovery rate (FDR) across the guardrails without inflating the primary's alpha. Emphasize that guardrails are typically non-inferiority checks, so use one-sided tests with a pre-specified margin, and describe a decision framework that prioritizes the primary while treating guardrail violations as blockers or escalation triggers.
Pro tip: Frame guardrails as 'canaries'—they are not meant to be optimized, but to detect harm; using a non-inferiority margin and a gatekeeping procedure (e.g., fixed-sequence) preserves power on the primary and avoids unnecessary multiplicity adjustments.
Define the primary metric as the sole confirmatory endpoint and guardrails as safety checks with non-inferiority hypotheses. State that guardrails are not for claiming improvement, only for detecting degradation.
Use a gatekeeping or hierarchical testing procedure: test the primary at full alpha first; only if significant, test guardrails with alpha control (e.g., Bonferroni or Holm) to preserve power on the primary. Alternatively, use a fixed-sequence procedure where guardrails are tested only after the primary wins.
Pre-specify acceptable degradation margins for each guardrail based on business impact, and power the experiment for the primary while ensuring guardrails have sufficient power to detect meaningful harm. Consider that guardrails often need larger sample sizes for non-inferiority.
If primary is significant and all guardrails pass non-inferiority, ship. If primary is significant but a guardrail fails, escalate for review or consider a follow-up experiment. If primary is not significant, do not test guardrails for superiority; only check for harm if pre-registered.
Discuss how the chosen procedure affects power and false positive rates, and provide sensitivity analyses (e.g., different margins, FDR vs FWER) to show robustness of conclusions.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Contamination biases ITT toward the null, attenuating the treatment effect estimate.
First, explain that contamination dilutes the treatment effect, biasing the ITT estimate toward the null. Then, describe how to correct for it using instrumental variables or by estimating the complier average causal effect (CACE), while noting the assumptions required.
Pro tip: Mention that while ITT is the gold standard for causal inference, it estimates the effect of assignment, not treatment receipt. For a more precise effect, consider CACE, but be transparent about its assumptions and limitations.
Explain that contamination in the control group dilutes the difference between treatment and control, biasing the ITT estimate toward zero (null effect).
Determine the proportion of control users exposed to treatment (10%) and assess how it reduces the observed effect size.
Use instrumental variables (IV) with random assignment as the instrument to estimate the Complier Average Causal Effect (CACE), or apply a simple adjustment if the exposure rate is known.
Check that the instrument is valid (random assignment, exclusion restriction) and that the exposure is accurately measured. Discuss potential violations.
Compare the corrected estimate (CACE) with the ITT estimate, and discuss the implications for decision-making, noting that ITT remains relevant for policy.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.