TikTok·Data Scientist·Technical Phone Screen
- You're launching a tipping feature on creator posts and need to design an A/B test that handles interference and supply-demand dynamics. Walk through your randomization unit and clustering strategy to control for cross-user and cross-post spillovers.
- How would you define eligibility and exposure rules for this experiment to prevent treatment contamination across different time zones, specifically US and Asia?
- Define the primary metric hierarchy for this tipping experiment, including guardrail metrics around retention, session length, abuse, and ad revenue cannibalization.
- How would you approach power analysis and experiment duration for this test, including ramp planning, sample ratio mismatch detection, and pre-registered stop rules?
- What variance reduction techniques would you apply here, and how would you handle inference given clustered randomization?
- If the tipping feature improves monetization metrics but causes a small but real drop in growth metrics, what decision framework would you use to decide whether to ship it?
“I went with creator-level randomization pretty quickly, reasoning that if a viewer sees a tipped post and then a non-tipped post from a different creator, you've already got contamination.”