Amazon·Data Scientist·Technical Phone Screen
- Walk me through how you would design, run, and analyze an A/B test end to end for a product change.
- How do you choose the unit of randomization and define what counts as an exposed user?
- How do you determine sample size, minimum detectable effect, and statistical power for your experiment?
- What data quality checks do you run before trusting your experiment results?
- How do you handle the multiple testing problem and what's your view on sequential peeking at results?
- How do you decide whether to launch based on experiment results? How do you weigh statistical significance against practical significance?
- What are the most common pitfalls in A/B testing, particularly around interference, noncompliance, and missing data?
“This is the kind of question that sounds manageable until you're 10 minutes in and realize you've only covered hypothesis framing and the interviewer is still waiting.”