I knew the mechanics well enough but stumbled a bit explaining the 'why' before the 'how'.
Start by explaining why the two-sample t-test is appropriate for comparing means of a continuous metric between two independent groups, then walk through the end-to-end process from defining hypotheses to interpreting results. Emphasize practical considerations like assumption checks, sample size, and potential pitfalls in an A/B testing context.
Pro tip: Mention that while the t-test is robust to moderate violations of normality with large samples (thanks to the CLT), you should still check for extreme skew or outliers and consider transformations or non-parametric alternatives if needed. Also, highlight the importance of pre-registering the analysis plan to avoid p-hacking.
Explain that the two-sample t-test compares means of a continuous metric between two independent groups (control and treatment), assuming approximately normal sampling distributions or large samples. It's appropriate because we're interested in the difference in average order value or session length.
State the null hypothesis (no difference in means) and alternative hypothesis (two-sided or one-sided). Clearly define the metric and the unit of randomization (e.g., user-level).
Verify independence, normality (or large sample size), and equal variances (or use Welch's t-test). Determine required sample size via power analysis (typically 80% power, 5% significance). Ensure random assignment and no interference between groups.
Calculate the t-statistic and p-value using the appropriate formula (pooled or Welch). Also compute confidence intervals for the difference in means to understand practical significance.
Compare p-value to significance level (e.g., 0.05). If significant, assess effect size and business impact. Consider multiple testing corrections if multiple metrics. Discuss limitations and next steps.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
This is where I actually felt more comfortable.
Start by explaining why user-level randomization is the default, then identify scenarios where it fails due to interference, spillover, or operational constraints. For each scenario, describe how switchback or geographic experiments address the issue, and discuss trade-offs like bias, variance, and feasibility.
Pro tip: Emphasize that switchback designs require careful time-series analysis to account for temporal trends, and geographic experiments need to handle spatial correlation and market-level heterogeneity. Mentioning these nuances shows depth beyond textbook answers.
Explain that user-level randomization assumes no interference between users and stable treatment effects. When these assumptions break, results can be biased.
Switchback is suitable when interference occurs within a shared resource over time (e.g., marketplace, supply chain) or when user-level randomization is infeasible due to operational constraints.
Geographic experiments are appropriate when interference spills over across users in the same region, when treatment affects an entire market (e.g., pricing, marketing campaigns), or when user-level randomization is impossible due to platform-wide changes.
Compare switchback and geographic designs in terms of bias, variance, and practical implementation. Mention the need for specialized statistical methods to account for temporal or spatial correlations.
Summarize when to choose each design based on the nature of interference, operational constraints, and the goal of the experiment.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.