Disney·Data Scientist·Technical Phone Screen
May 2026
Disney data scientist interview that went deep into experiment design, like really deep. Six questions all centered on one A/B test scenario for a paywall copy change, covering sample sizing, variance reduction, clustering, sequential testing, multiple comparisons, and noncompliance. Not a vibe check round at all.
- Starting from scratch: compute the per-arm sample size for a two-sided test with 80% power and alpha 0.05, given an 18% baseline conversion rate and a 7% relative minimum detectable effect. Walk through the formulas.
- If you apply CUPED using a pre-experiment covariate with R-squared of 0.25, how much does that reduce variance and what does the new required sample size look like?
- Adjust the sample size for household-level randomization using the design effect formula DE = 1 + (m-1) times ICC, where average household size is 1.8 and ICC is 0.06. Give the adjusted per-arm sizes with and without CUPED.
- How do O'Brien-Fleming spending boundaries change how Type I error is allocated across interim looks, and what does that mean practically for your experiment timeline and power?
- With 12 secondary metrics, how would you control false discovery rate at 10% and how do you communicate what a discovery actually means in that context?
- Given 10% noncompliance in treatment and 3% contamination in control, compute the ITT effect versus the complier average causal effect (CACE). How would you present both numbers to a product team without misleading them?
“This is the foundation question and I spent too long second-guessing whether the 7% was relative or absolute.” The rest of the author's notes on Data Scientist interview at Disney, Technical Phone Screen round, covers how they worked through the question, what the panel pushed back on, and what they would do differently.
View Post