Meta·Data Scientist·Technical Phone Screen·Senior

Senior

Jun 2026

Summary

Meta DS interview with a product experimentation case built around Instagram. The whole thing was one big scenario about a feature that hadn't launched yet, so you had to work entirely from proxy logic and experimental design. Pretty intense for a single question but it covered a lot of ground.

Questions Asked (3)

Instagram is thinking about adding a 'Show similar products' button. Before it launches, how would you convince a PM it's worth building, and what existing data would you use to estimate its potential impact on engagement?

Product Analytics & MetricsProduct Sense & Ideation

Author's notes

This part tripped me up more than I expected.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by framing the button as a hypothesis about user intent and business value, then outline a data-driven approach to validate it using existing logs and experiments. Focus on estimating impact through proxy metrics and causal inference methods, while acknowledging risks and trade-offs.

Pro tip: Emphasize that you would first check if similar products are already being viewed or clicked elsewhere in the app, as this reveals latent demand and provides a baseline for impact estimation.

1. Clarify the goal and hypothesis

Define what 'worth building' means (e.g., increased engagement, revenue) and state a clear hypothesis about how the button drives value.

2. Identify existing data sources

List relevant data such as user clickstream, product views, search queries, and A/B test results that can inform the estimate.

3. Estimate potential impact using proxies

Use historical data on similar features or user behaviors to model expected lift in engagement metrics (e.g., clicks, time spent).

4. Validate with a small-scale experiment

Propose a pilot A/B test to measure actual impact, and discuss how to size it and interpret results.

5. Communicate trade-offs and next steps

Summarize expected impact, confidence, and risks, and recommend whether to build, iterate, or abandon.

Key Points to Mention

Define success metrics (e.g., CTR, conversion rate, time spent) and guardrail metrics (e.g., user satisfaction, page load time).
Use existing data like clickstream logs, search queries, and product view history to find signals of demand for similar products.
Leverage past experiments or feature launches to estimate effect sizes and avoid common pitfalls like novelty effects.
Consider causal inference methods (e.g., propensity score matching, difference-in-differences) if a randomized experiment isn't immediately possible.
Quantify the opportunity size (e.g., number of users exposed, potential incremental clicks) to make a business case.
Acknowledge limitations and propose a phased approach: start with a small test, then scale if metrics are promising.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.

How would you design an experiment to decide whether to launch the 'Show similar products' button, including your randomization unit, what the control and treatment look like, and what guardrail metrics you'd set?

A/B Testing & ExperimentationProduct Analytics & Metrics

Author's notes

I went with user-level randomization first and then they pushed back on network effects, which, fair.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by clarifying the product goal and hypothesis, then outline a randomized controlled experiment with a clear unit of randomization (e.g., user-level). Define control and treatment experiences, primary and guardrail metrics, and explain how you would analyze and make a launch decision.

Pro tip: Emphasize that the randomization unit should align with the metric and interference level; for social features, consider cluster randomization to avoid contamination. Also, pre-register your metrics and decision criteria to avoid p-hacking.

1. Clarify goal and hypothesis

State the product goal (e.g., increase engagement or conversion) and formulate a testable hypothesis about how the 'Show similar products' button will impact user behavior.

2. Choose randomization unit and design

Select the randomization unit (e.g., user, session, or cluster) based on interference risk and metric sensitivity. Describe the control (no button) and treatment (button shown) experiences.

3. Define metrics and guardrails

Identify primary success metrics (e.g., click-through rate, conversion) and guardrail metrics (e.g., page load time, user satisfaction, revenue) to monitor for negative side effects.

4. Plan analysis and decision criteria

Specify statistical tests, power analysis, and decision rules (e.g., minimum detectable effect, significance level). Include how to handle multiple comparisons and segment analysis.

5. Consider practical constraints

Address potential pitfalls like novelty effects, seasonality, and sample size. Discuss rollout strategy if the test is successful.

Key Points to Mention

Randomization unit: user-level to avoid contamination, but consider cluster randomization if network effects exist.
Control and treatment: control sees no button, treatment sees the button; ensure consistent placement and functionality.
Primary metric: e.g., product detail page views or add-to-cart rate; guardrail metrics: page load time, bounce rate, revenue per user.
Statistical power: calculate required sample size and duration to detect a meaningful effect.
Analysis: use t-test or regression, check for novelty effects, and pre-register metrics.
Decision framework: launch if primary metric improves significantly without harming guardrails.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.

What criteria would tell you it's safe to roll out the feature to everyone?

A/B Testing & ExperimentationProduct Strategy

Author's notes

Short answer and I think I gave a short answer.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by defining the primary success metric and guardrail metrics, then outline the statistical and practical criteria for a full rollout. Emphasize that the decision should balance statistical significance, effect size, and business impact while ensuring no harm to user experience.

Pro tip: Mention that you would also consider the novelty effect and long-term holdout groups to ensure the effect persists beyond the initial launch period. This shows you think beyond the immediate experiment and understand Meta's scale and focus on sustained impact.

1. Define success and guardrail metrics

Identify the primary metric that measures the feature's success (e.g., engagement, revenue) and key guardrail metrics (e.g., user satisfaction, latency, crash rates) that must not degrade.

2. Check statistical significance and power

Ensure the experiment has enough power to detect a meaningful effect and that the results are statistically significant (e.g., p-value < 0.05) for the primary metric.

3. Assess practical significance and effect size

Evaluate whether the observed effect size is large enough to matter for the business, considering the cost of implementation and potential risks.

4. Evaluate guardrail metrics and segment analysis

Verify that guardrail metrics are not negatively impacted and check for heterogeneous treatment effects across key segments (e.g., new vs. existing users, regions).

5. Consider long-term impact and rollout strategy

Plan for a gradual rollout or holdout to monitor long-term effects, and ensure the feature aligns with strategic goals and user experience.

Key Points to Mention

Primary metric improvement with statistical significance (e.g., p-value < 0.05)
Guardrail metrics not showing significant negative impact
Effect size and confidence intervals to assess practical significance
Segment analysis to ensure consistent positive impact across key user groups
Novelty effect and long-term holdout to validate sustained impact
Business impact and alignment with product strategy

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.