← Meta Interview Insights

Meta·Data Scientist·Technical Phone Screen·Senior

Senior
May 2026

Summary

Meta data scientist interview with a meaty offline evaluation question that basically tests whether you know what to do when you can't just say 'run an experiment.' The kind of question that sounds straightforward until you realize how many angles they actually want you to cover.

Questions Asked (1)

Q1

A new recommendation feature is being considered for launch. Using only historical data, how would you assess whether it's worth releasing? Walk through the analyses, metrics, and assumptions you'd rely on. (Note: A/B testing is off the table as an answer.)

A/B Testing & ExperimentationProduct Analytics & MetricsTechnical Trade-offs
Author's notes

This one tripped me up a bit because my instinct was to reach for an experiment setup and they explicitly blocked that path.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Frame the problem as a causal inference challenge: define the counterfactual (what would have happened without the feature) and use historical data to estimate it via methods like propensity score matching, difference-in-differences, or synthetic control. Then translate the estimated impact into business metrics (e.g., CTR, engagement, revenue) and assess robustness through sensitivity analyses and assumptions checks.

Pro tip: Acknowledge that without randomization, you're relying on strong assumptions (e.g., no unmeasured confounders, parallel trends) and that the goal is to build a credible case, not prove causality. Show maturity by discussing how you'd validate assumptions and quantify uncertainty.

1. Define the causal question and estimand

Clarify what 'worth releasing' means: which metric(s) matter (e.g., CTR, watch time, revenue) and what is the target population. Specify the estimand, such as the average treatment effect on the treated (ATT) for users who would be exposed to the feature.

2. Identify a natural experiment or quasi-experimental design

Look for historical rollouts, gradual launches, or exogenous shocks that create variation in exposure. Consider methods like difference-in-differences, synthetic control, or instrumental variables if a natural experiment exists.

3. Estimate the counterfactual using observational methods

If no natural experiment, use propensity score matching, inverse probability weighting, or doubly robust estimation to create comparable treatment and control groups. Adjust for confounders and check covariate balance.

4. Quantify impact and uncertainty

Estimate the effect on key metrics with confidence intervals, and conduct sensitivity analyses (e.g., Rosenbaum bounds, placebo tests) to assess robustness to unmeasured confounding. Translate into business impact (e.g., incremental revenue).

5. Make a recommendation with caveats

Weigh the estimated benefits against costs and risks, and state the assumptions and limitations. Recommend release only if the evidence is strong and robust, or suggest a phased rollout with monitoring if uncertainty is high.

Key Points to Mention

  • Causal inference methods: propensity score matching, difference-in-differences, synthetic control, instrumental variables
  • Assumptions: no unmeasured confounding, parallel trends, positivity, SUTVA
  • Metrics: define primary and guardrail metrics (e.g., CTR, engagement, revenue, user satisfaction)
  • Sensitivity analysis: Rosenbaum bounds, placebo tests, negative controls
  • Business impact: translate effect sizes into incremental revenue or engagement
  • Limitations: observational data cannot prove causality; discuss uncertainty and potential biases

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.