Meta·Data Scientist·Technical Phone Screen·Senior

Senior

Apr 2026

Summary

Meta DS interview with a product experimentation question about rolling out restaurant recommendations in the News Feed. Pretty involved for a single question but it covered a lot of ground.

Questions Asked (1)

Meta is thinking about adding restaurant recommendations to the News Feed. Walk through how you'd design an experiment to evaluate this: what's the treatment and control, what metrics would you track, how do you think about sample size, and what would it take to actually ship this?

A/B Testing & ExperimentationProduct Analytics & MetricsProduct Sense & Ideation

Author's notes

This one sprawled in a way I didn't expect.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by clarifying the product goal and hypothesis, then define a clean treatment/control split (e.g., News Feed with restaurant recommendations vs. standard feed). Outline primary and guardrail metrics, explain sample size and power considerations, and finish with a ship/no-ship decision framework that includes long-term and ecosystem effects.

Pro tip: Meta cares deeply about ecosystem effects and long-term user value, so explicitly discuss guardrail metrics (e.g., user well-being, content diversity) and how you'd measure long-term impact beyond the experiment window.

1. Clarify goal and hypothesis

Restate the product goal (e.g., increase restaurant discovery and engagement) and form a testable hypothesis about how adding restaurant recommendations to News Feed will affect user behavior.

2. Define treatment and control

Specify the treatment (News Feed with restaurant recommendation module) and control (current News Feed without it), ensuring randomization at the user level and considering holdout groups for long-term measurement.

3. Select metrics

Choose primary success metrics (e.g., restaurant recommendation CTR, restaurant page visits, reservations), secondary metrics (e.g., overall engagement, time spent), and guardrail metrics (e.g., user reports, unfollows, well-being survey scores).

4. Determine sample size and duration

Estimate required sample size using power analysis (e.g., 80% power, 5% significance) based on expected effect size and baseline variance; account for novelty effects and seasonality by running for at least 2-4 weeks.

5. Analyze results and decide to ship

Evaluate statistical significance, practical significance, and guardrail metrics; consider long-term holdout results and qualitative feedback before making a ship/no-ship recommendation.

Key Points to Mention

Randomization unit (user-level) and potential network effects (e.g., friends seeing recommendations)
Primary metric: restaurant recommendation click-through rate or reservation conversions
Guardrail metrics: user well-being, content diversity, and negative feedback
Sample size calculation: power, MDE, baseline variance, and multiple testing correction
Long-term holdout to measure sustained impact and avoid novelty effects
Ship criteria: statistically significant lift in primary metric without harming guardrails, plus positive long-term signal

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.