Meta·Data Scientist·Technical Phone Screen
- A multi-account product currently ranks accounts by most recent visit. The team wants to switch to ranking by highest notification count instead, with recency as a tiebreaker. Using historical session-level logs, how would you assess whether this change is likely to be beneficial before running an experiment?
- Design an A/B test to measure the causal impact of the new notification-based account ranking. Walk through the eligible population, randomization unit, primary and secondary metrics, guardrails, power and MDE considerations, and any key segments or edge cases.
“I started talking about correlational analysis between notification rank position and account selection rate, which felt right, but I kept second-guessing myself on the selection bias piece.”