This question looks like a standard experiment design prompt until you realize how many layers they actually want you to cover.
Start by clarifying the product goal and the specific hypothesis for the short-video feed feature, then systematically walk through each experimental design component (unit, randomization, metrics, sample size, guardrails). Emphasize trade-offs and practical considerations like novelty and seasonality, and conclude with a clear decision framework for calling the test.
Pro tip: Meta often uses cluster randomization (e.g., by user or by social graph) to avoid interference in feed experiments; mention this and discuss how you'd measure and mitigate network effects. Also, propose a pre-registered analysis plan with sequential testing or a fixed horizon to avoid peeking and p-hacking.
Articulate a clear, testable hypothesis (e.g., short-video feed increases engagement) and define what success looks like in terms of primary and secondary metrics. Align with stakeholders on the minimum detectable effect (MDE) and business impact.
Decide on the randomization unit (e.g., user, session, or cluster) based on interference risk and metric sensitivity. For Meta, user-level randomization is common, but consider cluster randomization if social spillover is a concern. Ensure proper randomization and balance checks.
Identify primary (e.g., time spent, video views) and secondary metrics (e.g., shares, comments), plus guardrail metrics (e.g., user reports, unfollows). Use power analysis to estimate required sample size and duration, accounting for variance and MDE.
Plan for novelty effects by running the test long enough to observe stabilization and analyzing early vs. late periods. Mitigate seasonality by using a control group, considering holdout periods, or using techniques like CUPED to adjust for pre-experiment trends.
Pre-define stopping rules: either a fixed horizon based on power analysis or sequential testing with alpha spending. Monitor guardrails continuously and stop early if harm is detected. At the end, analyze results with appropriate statistical tests and consider practical significance.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.