This is where I spent most of my mental energy.
Start by acknowledging the interference problem in social networks and the constraints (10% cap, skewed degrees). Then propose a randomization scheme that balances interference control with practical feasibility, such as cluster randomization on the graph or ego-network randomization, and discuss trade-offs like bias, variance, and power.
Pro tip: Emphasize that the choice of experimental unit should align with the interference mechanism: if interference is mainly through close connections, randomize at the cluster level (e.g., communities) to contain spillover; if it's global, consider a switchback or time-based randomization. Also, mention that with a 10% cap, you might need to use a biased but low-variance design like graph cluster randomization with a small number of clusters, and correct for bias via post-stratification.
Clarify the interference mechanism (e.g., spillover through social ties) and the constraints: 10% treatment cap and skewed degree distribution. Discuss how these affect the choice of experimental unit.
Consider units like users, ego-networks, communities, or time-based switchbacks. For each, assess how well they contain interference, their feasibility under the 10% cap, and their statistical properties (bias, variance, power).
Select a scheme (e.g., cluster randomization on communities) and explain why it balances interference reduction with the cap and skew. Discuss how to handle the skewed degree distribution, e.g., by weighting or stratifying by degree.
Outline how to implement the randomization (e.g., graph partitioning algorithms) and analyze results (e.g., cluster-robust standard errors, bias correction). Mention potential pitfalls like dilution and network effects.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
I knew what spillover meant conceptually but defining it as a formal estimand with exposure conditions was harder than I expected.
Start by defining the causal estimands in a potential outcomes framework, clearly distinguishing direct, spillover, and total effects. Then introduce an exposure model (e.g., partial interference with a known exposure mapping) that makes these effects identifiable, and discuss assumptions and estimation strategies. Finally, connect to practical A/B testing scenarios at Meta, such as network effects in social experiments.
Pro tip: Emphasize that spillover effects are only identifiable under specific exposure models, and that ignoring them can lead to biased estimates of the total effect. Mention that in practice, cluster randomization or graph-based exposure models are often used to mitigate interference.
Formally define the direct effect as the difference in outcomes for an individual under treatment vs. control, holding the treatment assignment of others fixed. Define spillover effect as the difference in outcomes for an individual due to changes in the treatment assignment of others, holding their own treatment fixed. Define total effect as the difference between the outcome when an individual is treated and others are treated in some way vs. when the individual is untreated and others are untreated.
Describe an exposure model that maps the treatment assignment vector to an individual's exposure. For example, partial interference with a known exposure mapping (e.g., the proportion of treated neighbors). State assumptions like consistency, no unmeasured confounding, and correct specification of the exposure model that make the estimands identifiable from observed data.
Explain how to estimate these effects using methods such as inverse probability weighting, outcome regression, or machine learning under the exposure model. Mention challenges like interference and the need for cluster-robust standard errors.
Relate the concepts to real-world A/B testing at Meta, where interference is common due to social networks. Discuss strategies like cluster randomization, graph cluster randomization, or using exposure models to adjust for spillovers in experiments.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Start by clarifying the experimental design (e.g., cluster randomization, user-level randomization with clustered data) and the estimand (e.g., average treatment effect). Then describe the point estimate (difference in means or weighted average) and how to compute standard errors using cluster-robust or randomization inference, explicitly stating assumptions and consequences of violations.
Pro tip: Emphasize that cluster-robust SEs are asymptotic and rely on a sufficient number of clusters; with few clusters, consider bootstrap or randomization inference. Also, mention that randomization inference is exact under the sharp null and can be more robust when clusters are few or unbalanced.
Identify the randomization unit (e.g., user, cluster) and the outcome metric. Define the causal estimand (e.g., ATE) and the point estimate (e.g., difference in means or regression coefficient).
Calculate the point estimate as the difference in average outcomes between treatment and control, or via regression with treatment indicator. If clusters are unequal, consider weighting by cluster size.
For cluster-robust SEs, use the sandwich estimator with cluster-level sums of scores. For randomization inference, permute treatment assignment at the cluster level and compute the distribution of the test statistic under the sharp null.
For cluster-robust, use normal approximation: estimate ± 1.96 * SE. For randomization inference, use the 2.5th and 97.5th percentiles of the permutation distribution of the estimator (or invert the test).
List key assumptions: clusters are independent, large number of clusters for asymptotic validity, correct model specification for cluster-robust; for randomization inference, sharp null and exchangeability. Discuss what breaks: few clusters lead to undercoverage; violations of independence (e.g., interference) bias SEs; model misspecification affects cluster-robust.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Okay so I actually like this kind of question because it's just math.
First, compute the required sample size for an individually randomized design using the baseline mean, standard deviation, and target relative lift to achieve 80% power. Then, adjust for clustering by calculating the design effect (1 + (average cluster size - 1) * ICC) and multiply the individual sample size by this factor to get the required number of users. Finally, divide by the average cluster size to determine the number of clusters needed.
Pro tip: Always clarify whether the sample size refers to users or clusters, and mention that the design effect inflates the variance, so you need more clusters to maintain power. Also, note that the ICC is typically small in online experiments, but even a small ICC can substantially increase sample size when clusters are large.
Calculate the required sample size per arm assuming no clustering, using the formula for comparing means: n = 2 * (Z_alpha/2 + Z_beta)^2 * SD^2 / (delta)^2, where delta is the absolute lift (baseline mean * relative lift).
Use the design effect formula: DE = 1 + (m - 1) * ICC, where m is the average cluster size (120) and ICC is 0.02. This quantifies the variance inflation due to clustering.
Multiply the individual sample size by the design effect to get the required number of users (or observations) under clustering. This is the effective sample size needed.
Divide the adjusted sample size by the average cluster size (120) to determine the number of clusters required per arm. Round up to the nearest whole number.
Verify that the numbers make sense (e.g., design effect >1, sample size larger than individual case) and discuss practical implications such as feasibility and potential for unequal cluster sizes.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Cut-edge exposure rate was my first one, treated-producer content share seen by control users was second.
Start by framing the goal: to ensure experiment validity by detecting leakage or interference. Then, for each diagnostic check, describe the method, the expected outcome under no leakage/interference, and the action plan if the check fails. Emphasize a systematic, data-driven approach to root cause analysis and mitigation.
Pro tip: Prioritize checks that are easy to implement and interpret, such as Sample Ratio Mismatch (SRM) and pre-experiment covariate balance, as they often reveal subtle issues. Also, consider the trade-offs between sensitivity and specificity of each check.
Compare the observed ratio of users in treatment and control groups to the expected ratio using a chi-squared test. A significant deviation indicates potential leakage or assignment issues.
Compare key pre-experiment metrics (e.g., age, tenure, past behavior) between groups using statistical tests or effect sizes. Imbalance suggests interference or non-random assignment.
Analyze metrics that should not be affected by the treatment (e.g., user demographics, pre-period behavior). Significant differences indicate leakage or interference.
For each failed check, investigate root causes (e.g., logging errors, cross-contamination, network effects). Mitigate by fixing bugs, re-randomizing, or using techniques like cluster randomization or isolation.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Went with alpha-spending using an O'Brien-Fleming boundary because it's conservative early and I've actually used it before.
Start by acknowledging the need for a sequential testing method that allows for frequent interim looks while controlling Type I error. Recommend the alpha spending approach (e.g., O'Brien-Fleming or Pocock) or group sequential methods, and specify how you would set stopping boundaries and continuation rules. Emphasize the trade-offs between early stopping and error control, and mention practical considerations for implementation at Meta's scale.
Pro tip: Mention that with looks every two days, you need to account for the number of looks and the timing, and that alpha spending functions like O'Brien-Fleming are often preferred for their conservative early stopping, which is useful when metrics need time to stabilize. Also, note that you would pre-register the analysis plan to avoid p-hacking.
Select a group sequential design with alpha spending, such as O'Brien-Fleming or Pocock boundaries, or a continuous monitoring approach like always valid inference. Justify based on the need for frequent looks and Type I error control.
Specify how alpha is allocated across interim looks. For example, O'Brien-Fleming spends little alpha early and more later, while Pocock spends equally. This determines the stopping boundaries.
Establish boundaries for efficacy and futility at each look. If the test statistic crosses the efficacy boundary, stop for success; if it crosses futility, stop for no effect; otherwise continue to the next look.
Use software (e.g., gsDesign in R) to compute exact boundaries based on the spending function, number of looks, and information fractions. Adjust p-values and confidence intervals for sequential monitoring.
Discuss handling of uneven look intervals, missing data, and multiple metrics. Mention that you would pre-register the plan and monitor for any deviations.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
This one felt like a trick question at first because the setup is basically 'what do you do when you can't fix the design.' I mentioned CUPED with pre-period to reduce variance, exposure-weighted IV as a way to isolate the complier effect, and sensitivity bounds to show a range of plausible ATEs under different interference assumptions.
Acknowledge the constraint and focus on post-hoc methods to quantify and bound bias from ignoring clustering. Use cluster-robust variance estimation, sensitivity analysis, and conservative adjustments to produce valid confidence intervals.
Pro tip: Emphasize that while you can't eliminate bias post-hoc, you can bound it and be transparent about assumptions. Show you understand the trade-off between validity and precision.
Assess the degree of clustering in the data by computing intra-cluster correlation (ICC) for key metrics. This quantifies how much user-level randomization violates independence.
Apply cluster-robust standard errors (e.g., CR2 or bootstrap) at the cluster level to account for within-cluster correlation. This yields conservative confidence intervals.
Perform sensitivity analysis by varying assumptions about cluster effects (e.g., using different ICC values) to see how conclusions change. Report the range of treatment effects.
Use methods like the delta method or variance inflation factors to adjust point estimates and intervals. Consider worst-case bounds by assuming maximum plausible correlation.
Clearly document the limitations and present results with conservative intervals. Recommend follow-up experiments with proper randomization if possible.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.