Meta·Data Scientist·Onsite - Multi Round
- How would you define the exposure unit and the estimand for an A/B test of a Group Call feature where network interference is expected? For example, should you target intent-to-treat at the cluster level or a per-user average treatment effect?
- How would you construct clusters from a social graph for this experiment? Walk through how you'd weight edges for strong ties, choose a clustering rule like connected components after thresholding, and prevent cluster overlap.
- If you randomize at the cluster level, how do you handle users who appear on edges between treatment and control clusters? What role do holdout buffers or partial saturation designs play?
- What primary metric and two guardrail metrics would you choose for this feature? Critically assess 'time spent per user per day' as a success metric, including its pros, cons, and manipulation risks, and propose at least one alternative primary metric.
- Compute the design effect for cluster randomization given an average cluster size of 20 and an ICC of 0.05. How does the design effect change if the average cluster size doubles but the ICC halves?
- If you ignored network effects and randomized by individual user instead of cluster, under what conditions would the naive treatment effect estimate be biased downward versus upward? Give intuition for positive and negative spillovers and a real-world example of each.
“This is where I started to feel the ground shift.”