TikTok·Data Scientist·Technical Phone Screen
- You ran an A/B test on a video length change and got a non-significant result on your primary metric. Baseline completion rate is 22% and the product team wants at least a 5% relative lift to matter. Walk through how you'd calculate the required per-arm sample size for 80% power at a two-sided alpha of 0.05, and explain your assumptions.
- How would you apply variance reduction techniques like CUPED using pre-period completion data, and if your covariate explains 25% of the outcome variance, what's the practical effect on your test's power?
- Given the non-significant result, should you keep collecting data, stop the experiment, or switch to a Bayesian approach with a utility-based stopping threshold? Defend your choice.
- If you're tracking multiple related outcomes alongside your primary metric, how do you handle the multiplicity problem and what corrections would you apply?
- How would you design a sequential testing approach, like an O'Brien-Fleming spending function, that allows you to peek at results early without inflating your Type I error rate?
“This part I actually felt okay about.”