I started with randomization unit and felt okay there, user-level made sense to avoid exposure bleed.
Start by defining clear success metrics (e.g., CTR, engagement, diversity) and then outline a randomized controlled experiment with proper randomization unit and sample size calculation. Emphasize the importance of avoiding network effects and ensuring valid inference through techniques like cluster randomization or switchback testing.
Pro tip: At Meta, where social influence is strong, consider using cluster randomization by user or by social graph to mitigate interference, and always run an A/A test beforehand to validate your experimentation setup.
Clearly state the hypothesis (e.g., new algorithm increases engagement) and select primary and guardrail metrics (e.g., CTR, time spent, diversity, user satisfaction).
Decide whether to randomize at user, session, or cluster level (e.g., by social graph) to avoid interference; consider switchback or geo-based randomization if needed.
Determine required sample size based on expected effect size, power (80%), significance level (5%), and variance; account for network effects if present.
Launch the experiment, monitor for SRM (sample ratio mismatch), and ensure data quality; avoid peeking or early stopping unless using sequential testing.
Analyze primary and guardrail metrics, check for heterogeneous treatment effects, and make a ship/no-ship decision based on statistical and practical significance.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.