I got through the basic A/B setup fine but fumbled when they pushed on variance reduction.
Start by defining the experimental unit (restaurant location) and the treatment (surcharge applied to all orders from selected locations). Then outline the key metrics (profitability, order volume, customer satisfaction) and how to measure them, followed by methods to increase statistical power such as stratification, blocking, and using historical data as covariates.
Pro tip: Consider the ethical and business implications of the surcharge: it may disproportionately affect low-income customers, and the least profitable locations might already have low volume, making it hard to detect effects. Propose a phased rollout or a switchback design to mitigate these issues.
Randomly assign restaurant locations to treatment (surcharge) and control groups. Ensure the unit of randomization is the restaurant location to avoid contamination.
Identify primary metrics: restaurant profitability (e.g., contribution margin), order volume (number of orders), and customer satisfaction (e.g., ratings, NPS). Also consider guardrail metrics like overall platform revenue.
Calculate the required number of locations per group to detect a meaningful effect size with desired power (e.g., 80%) and significance level (e.g., 5%). Use historical data to estimate variance.
Use techniques like stratification (e.g., by location profitability, order volume), blocking, paired designs (e.g., switchback), and include covariates (e.g., historical sales) in analysis to reduce noise.
Use appropriate statistical tests (e.g., t-test, regression) to compare metrics between groups. Check for heterogeneous treatment effects and ensure results are robust.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.