Roblox·Data Scientist·Technical Phone Screen
- For an experiment testing increased ad load and a higher floor price in a mobile ads marketplace, how would you choose the unit of randomization, and how would you handle interference risks like auction dynamics or supply constraints?
- What primary KPI and guardrail metrics would you use to make a go/no-go decision, and what are your exact success and stop-loss thresholds?
- Walk through the power analysis for this experiment. What sample size, traffic split, and duration do you need for 90% power at alpha 0.05 to detect a 10% lift in daily revenue per user?
- How would you control for bias in this experiment? Cover pre-experiment covariate adjustment, seasonality, and your approach to sequential versus fixed-horizon testing, including how you'd handle multiple variants.
- How would you detect advertiser-side cannibalization and shifts in clearing prices when running this experiment? What diagnostics would you look at?
“This is where I stumbled first.”