Disney·Data Scientist·Technical Phone Screen
- Define the primary decision metrics and guardrails for evaluating a homepage ranking model rollout, and set concrete thresholds for each.
- Without a configured holdout group, how would you use Android users as a difference-in-differences control for the iOS rollout? What assumptions does this require and how do you test them?
- How would you design a geo-based synthetic control within iOS as an alternative identification strategy, and what diagnostics would you run?
- The rollout window includes a major content premiere and a payments outage. How do you adjust your analysis to account for these events, and what fixed effects or controls do you include?
- Quantify the business impact of the observed watch-time decline with confidence intervals, and estimate the weekly revenue effect across both ad and subscription revenue paths.
- Outline an immediate remediation plan for the rollout and a proper experiment design for the following month, including holdouts, pre-registration, variance reduction, and exposure logging.
“I led with watch-time per session as the north star, which felt right, but I fumbled the guardrails a bit.”