Choose a single experiment where you owned the design and analysis, and narrate it as a story: business problem, hypothesis, design choices, execution, and results. Emphasize the decisions you made to ensure validity (randomization, power, guardrails) and how you translated findings into product action.
Pro tip: Quantify the business impact (e.g., lift in key metric, revenue, or user retention) and briefly mention a trade-off or limitation you navigated—this shows you think like an owner, not just an analyst.
Describe the business problem, why it mattered, and the specific, testable hypothesis. Tie it to a clear primary metric and any guardrail metrics.
Explain your choices: unit of randomization, control/treatment, sample size and power calculation, duration, and how you handled potential confounders or network effects.
Cover implementation details, data quality checks, and how you monitored for SRM, novelty effects, or early stopping rules.
Describe the statistical methods used (e.g., t-test, CUPED, sequential testing), how you handled multiple comparisons, and what the results showed.
Explain the recommendation you made, the business impact, and how you communicated uncertainty and next steps to stakeholders.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.