TikTok·Data Scientist·Technical Phone Screen
- You're launching an A/B test for a new recommendation algorithm meant to increase content exploration. Walk through how you'd design and run that experiment to decide whether to launch.
- How would you define metrics for exploration, long-term user satisfaction, and creator ecosystem health in this context?
- The experiment window is too short to observe long-term impact. How do you analyze the data you have and make a launch recommendation anyway?
- Exploration metrics go up but like rate drops. Do you ship?
- How do you stop the model from just surfacing random irrelevant content in the name of diversity?
- How would you measure whether users actually return to creators they discovered through the new algorithm?
“This is where I spent most of my time and honestly fumbled the randomization discussion a bit.”