Amazon·Data Scientist·Technical Phone Screen·Senior

Senior

Jul 2026

Summary

Amazon data scientist interview with a meaty A/B testing design question. The whole session basically lived or died on how deep you could go on experimentation fundamentals, not just the surface stuff.

Questions Asked (1)

An e-commerce company is considering offering a 10% discount to boost conversions. Walk through how you would design and analyze an A/B test for this: hypotheses, metrics, sample size, experiment setup, and how you'd interpret the results.

A/B Testing & ExperimentationProduct Analytics & MetricsPricing & Monetization

Author's notes

This looked like a clean textbook question until I actually started answering it.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by framing the business goal and defining clear, testable hypotheses for the discount's impact on conversion and profit. Then systematically walk through the experiment design, including metric selection, sample size calculation, randomization, and analysis plan, emphasizing trade-offs and potential pitfalls. Conclude with how you would interpret results, considering statistical significance, practical significance, and business implications.

Pro tip: Always tie metrics back to the company's north star (e.g., long-term profitability) and mention guardrail metrics to catch unintended consequences like margin erosion or cannibalization. Show you understand that a statistically significant lift in conversion might not be worth it if it hurts profit.

1. Define Hypotheses and Success Metrics

State the null and alternative hypotheses (e.g., H0: discount does not affect conversion; H1: discount increases conversion). Choose primary metric (conversion rate) and secondary/guardrail metrics (average order value, profit margin, return rate, customer lifetime value).

2. Determine Sample Size and Duration

Calculate required sample size using power analysis (e.g., 80% power, 5% significance) based on minimum detectable effect (MDE) that is practically significant. Estimate daily traffic to determine test duration, ensuring it covers full business cycles (e.g., weekdays/weekends).

3. Design Experiment Setup

Randomly assign users to control (no discount) and treatment (10% discount) groups, ensuring proper randomization and avoiding contamination. Consider stratification by key covariates (e.g., geography, user tenure) and decide on triggering event (e.g., on product page view).

4. Analyze Results

After data collection, check for sample ratio mismatch (SRM) and validity. Compute lift in primary metric with confidence intervals and p-values. Perform subgroup analyses and check guardrail metrics for unintended effects.

5. Interpret and Recommend

Assess statistical and practical significance: is the lift in conversion worth the discount's impact on profit? Consider long-term effects and potential cannibalization. Provide a clear recommendation (launch, iterate, or abandon) with caveats.

Key Points to Mention

Hypothesis testing framework: null vs. alternative, one-tailed vs. two-tailed test
Primary metric: conversion rate; guardrail metrics: profit margin, AOV, return rate, customer lifetime value
Sample size calculation: power, significance level, minimum detectable effect, and traffic considerations
Randomization unit (user-level) and potential pitfalls like network effects or spillover
Analysis techniques: t-test, regression adjustment, sequential testing if peeking
Business interpretation: statistical significance vs. practical significance, long-term impact, and ROI

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.