← Snowflake Interview Insights
This is where I spent most of my mental energy.
First, compute the required sample size per variant using the standard two-proportion z-test formula for a 2% baseline and 5% relative lift (2.1% absolute), with 90% power and alpha 0.05. Then adjust for 5% bot traffic and 1% sample ratio mismatch by inflating the required sample size, and finally divide by the effective daily traffic per variant to get the minimum test duration in days.
Pro tip: Always clarify whether the sample size formula uses a one-sided or two-sided test; for a two-sided test at alpha 0.05, use z_alpha/2 = 1.96, and for 90% power, z_beta = 1.28. Also, remember that sample ratio mismatch (SRM) reduces the effective sample size per variant, so you need to inflate the total sample size by 1/(1-SRM) to maintain power.
Use the formula n = (z_alpha/2 * sqrt(2*p*(1-p)) + z_beta * sqrt(p1*(1-p1)+p2*(1-p2)))^2 / (p2-p1)^2, where p=0.02, p1=0.02, p2=0.021, z_alpha/2=1.96, z_beta=1.28. This yields approximately n ≈ 383,000 per variant.
Since 5% of traffic is bots, only 95% of users are real. Inflate the required sample size by dividing by 0.95: n_adjusted = n / 0.95 ≈ 403,158 per variant.
A 1% SRM means the actual split may deviate from 50/50, reducing effective sample size. Inflate further by dividing by (1 - 0.01) = 0.99: n_final = n_adjusted / 0.99 ≈ 407,230 per variant.
Total daily traffic is 1M users, so per variant it's 500,000. After removing 5% bots, effective daily traffic per variant is 500,000 * 0.95 = 475,000. Minimum days = n_final / 475,000 ≈ 0.857 days, so round up to 1 day. However, consider weekly seasonality and novelty effects; often a minimum of 1-2 weeks is recommended.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
I talked through bounce rate, p95 latency, crash rate, revenue per user as guardrails.
Start by defining guardrail metrics that protect user experience and business health, then explain how sequential monitoring with alpha-spending allows early stopping while controlling Type I error. Emphasize the trade-offs and practical implementation, showing you understand both statistical rigor and product impact.
Pro tip: Mention that guardrails should be pre-registered and monitored with the same statistical rigor as primary metrics, and that alpha-spending functions like O'Brien-Fleming are often preferred for their conservative early stopping boundaries.
Select metrics that capture potential negative side effects of the experiment, such as latency, error rates, user engagement, revenue, or retention. Ensure they are aligned with business goals and user experience.
Specify how often you will monitor the guardrails (e.g., daily) and the thresholds for concern. Decide whether to use sequential testing or fixed-horizon analysis, and pre-register the plan.
Use an alpha-spending function (e.g., O'Brien-Fleming, Pocock) to allocate the overall alpha across interim analyses, ensuring the family-wise error rate remains at the desired level (e.g., 5%).
At each interim look, compute the test statistic and compare it to the adjusted boundary from the alpha-spending function. Stop early if a guardrail is breached or if the primary metric shows significant harm/benefit, depending on the goal.
If early stopping is triggered, investigate the cause, communicate findings to stakeholders, and decide whether to modify or halt the experiment. Document the decision process for future reference.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Pre-experiment CTR per user as the covariate, pretty standard.
Start by explaining CUPED's core idea: using a pre-experiment covariate to reduce variance without introducing bias. Then, address novelty effects and day-of-week variation by choosing covariates that capture these patterns, such as pre-period metrics adjusted for day-of-week. Finally, detail validation steps like checking covariate balance, ensuring exogeneity, and comparing variance reduction with bias checks.
Pro tip: Emphasize that CUPED is not a magic fix; it requires careful covariate selection and validation. Mention that using multiple covariates (e.g., pre-period metric and day-of-week indicators) can be more effective, but always validate that they are uncorrelated with treatment assignment.
Briefly describe how CUPED uses a pre-experiment covariate to reduce variance by adjusting the outcome metric. Highlight that the covariate must be unaffected by the treatment.
Discuss how novelty effects can be mitigated by using a pre-period covariate that captures stable user behavior before the experiment. For day-of-week variation, include day-of-week indicators or use a covariate that is averaged over full weeks to avoid confounding.
Propose using the pre-experiment value of the same metric (e.g., average daily sessions in the 2 weeks prior) as the primary covariate. Additionally, include day-of-week fixed effects or a covariate like 'same day-of-week average' to account for weekly patterns.
Check that the covariate is balanced between treatment and control (e.g., t-test). Ensure the covariate is not affected by the treatment (exogeneity). Compare variance reduction via the ratio of variances, and use placebo tests (e.g., A/A tests) to confirm no bias is introduced.
Acknowledge that CUPED may not fully address novelty if the effect changes over time. Mention that day-of-week variation might require stratification or time-based covariates. Emphasize that validation is crucial to avoid bias.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Start by explaining the mechanisms to prevent interference and contamination at the user level, such as randomization, isolation, and monitoring. Then discuss when user-level bucketing is insufficient and alternative designs like geo-based or holdout are needed, focusing on trade-offs and use cases. Emphasize the importance of choosing the right design based on the experiment's goals and constraints.
Pro tip: Mention that even with user-level bucketing, network effects and shared resources can cause interference, so consider using cluster randomization or switchback designs. Also, highlight that geo-based designs can suffer from confounding due to regional differences, so matching or stratification is key.
Clarify what interference (spillover effects) and contamination (treatment leakage) mean in the context of experiments. Explain how they can bias results.
Describe techniques like consistent hashing, isolation of experiment environments, and monitoring for cross-arm contamination. Mention the importance of randomization unit and analysis unit alignment.
Discuss scenarios where user-level bucketing is insufficient, such as when there are network effects, shared resources, or when the treatment affects entire groups (e.g., pricing changes, UI changes that affect all users).
Explain when to consider geo-based designs (e.g., when treatment is at a geographic level, or to avoid spillover) and holdout designs (e.g., to measure long-term effects or when you want a control group that is completely untreated). Discuss trade-offs like reduced power, confounding, and longer duration.
Summarize criteria for choosing between user-level, geo-based, or holdout designs, such as the nature of the treatment, the presence of interference, and the goals of the experiment.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Post-hoc power check using observed variance rather than assumed variance, then subgroup breakdowns by things like user tenure or device type to catch HTE.
Start by explaining how you validate the experiment's statistical power using observed effect size and variance, then discuss methods for detecting heterogeneous treatment effects (HTEs) via subgroup analysis and interaction tests. Finally, outline a decision framework for ramping to 100% that balances statistical significance, practical significance, and risk mitigation.
Pro tip: Emphasize that you pre-register your analysis plan and use sequential testing or Bayesian methods to avoid peeking, and always consider the cost of false positives vs. false negatives when deciding to ramp.
Calculate the observed power using the actual sample size, effect size, and variance from the experiment. Compare it to the planned power (e.g., 80%) to ensure the experiment was adequately powered.
Perform subgroup analyses across pre-defined cohorts (e.g., user segments, geographies, device types) and test for interactions between treatment and cohort variables. Use methods like Causal Forests or meta-analyses to quantify HTEs.
Evaluate the primary metric's p-value and confidence interval, and consider the minimum detectable effect (MDE) and business impact. Ensure the effect is both statistically significant and practically meaningful.
Review guardrail metrics (e.g., latency, error rates, user satisfaction) to ensure no negative side effects. If HTEs show harm in certain cohorts, consider targeted rollouts or mitigation strategies.
Based on power, HTEs, significance, and guardrails, make a data-driven decision. If positive and safe, ramp gradually while monitoring; if uncertain, extend the experiment or run a follow-up.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.