Meta·Data Scientist·Technical Phone Screen·Senior

Senior

May 2026

Summary

Meta DS interview focused on experiment design for a new video ad format. One meaty question that covered a lot of ground, felt more like a product case than a pure stats problem.

Questions Asked (1)

Design an experiment to evaluate the effectiveness of a new video ad format. Walk through your choice of randomization unit, primary and secondary metrics, how you'd think about sample size and statistical power, and what success looks like. Also, if the primary metric doesn't move significantly, what do you do next?

A/B Testing & ExperimentationProduct Analytics & MetricsProduct Strategy

Author's notes

This one sprawled in ways I didn't fully anticipate.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Structure your answer by walking through the experiment design sequentially: randomization unit, metrics, power analysis, success criteria, and follow-up plan. Emphasize trade-offs and practical considerations, especially for a large-scale platform like Meta. Conclude with a clear decision framework for interpreting results and next steps.

Pro tip: Mention that you would pre-register the experiment and define guardrail metrics to catch unintended negative effects, showing rigor and business acumen. Also, discuss how you'd handle network effects or interference if the ad format could spill over between users.

1. Choose Randomization Unit

Decide whether to randomize at user, session, or ad level based on the ad format's delivery and potential interference. For Meta, user-level randomization is common to avoid contamination, but consider if the format is shown across sessions.

2. Define Metrics

Select a primary metric that directly measures the ad format's effectiveness (e.g., click-through rate, conversion rate, or video completion rate). Include secondary metrics for engagement, brand lift, and guardrails like user experience or revenue impact.

3. Plan Sample Size and Power

Calculate required sample size using expected effect size, baseline metric, desired power (typically 80%), and significance level (5%). Consider duration to account for novelty effects and seasonality.

4. Define Success Criteria

Specify what success looks like: a statistically significant improvement in the primary metric without degradation in guardrails. Also consider practical significance and business impact.

5. Plan for Non-Significant Results

If primary metric doesn't move, analyze secondary metrics, segment results, check for novelty effects, and consider qualitative feedback. Decide whether to iterate, run a longer test, or abandon the format.

Key Points to Mention

Randomization unit trade-offs: user-level avoids interference but may dilute effect if ad exposure is limited; session-level can capture immediate impact but risks contamination.
Primary metric should align with business goal (e.g., ad recall, conversion) and be sensitive to change; secondary metrics provide context and guardrails.
Power analysis requires assumptions about baseline rate, minimum detectable effect, and variance; use tools like power calculators or simulations.
Success is not just statistical significance but also practical significance and positive ROI; consider confidence intervals and effect size.
If no significant effect, check for implementation issues, segment by user demographics or behavior, and consider if the test was underpowered.
Always include guardrail metrics (e.g., user engagement, revenue) to ensure no negative side effects.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.