Meta·Data Scientist·Technical Phone Screen
- How would you choose the randomization unit for an A/B test on a hashtag recommender, and how does cross-session interference affect that choice?
- How would you structure the treatment and control groups, plan the rollout ramp, and prevent cross-over or contamination in this experiment?
- Define a primary success metric and at least three guardrail metrics for this hashtag recommender experiment.
- Walk through the sample size calculation for this experiment: baseline follow rate 4%, two-sided alpha 0.05, power 0.80, minimum detectable effect of +5% relative, with user-level randomization and an intraclass correlation of 0.02 across repeated sessions. Show the formulas and adjustments.
- What bias controls would you put in place for this experiment, including variance reduction techniques, novelty effect handling, sequential monitoring, and network interference diagnostics?
“This is where I stumbled first.”