Meta·Data Scientist·Onsite - Multi Round·Senior

Senior

Jul 2026

Summary

Brutal Meta DS onsite. One massive multi-part causal inference question covering everything from metric definition to Rosenbaum bounds. The kind of question where you realize halfway through that you've been answering a different version of it in your head.

Questions Asked (1)

Among Meta Quest users, those who use social features are hypothesized to be more regularly engaged than those who use game features. Given ~8 weeks of activity data, design a full analysis plan: define your primary engagement metric and guardrails, specify a randomized experiment if possible, propose an observational fallback with causal inference methods, compute the required sample size for a two-proportion z-test (baseline p0=0.35, MDE=+3pp, alpha=0.05, power=0.80), describe your estimator and how you handle multiple comparisons, detail at least five validity threats and robustness checks, and define a decision rule plus a communication plan for stakeholders.

A/B Testing & ExperimentationProduct Analytics & MetricsData Modeling

Author's notes

This was basically seven questions stapled together and presented as one.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by defining a clear primary engagement metric (e.g., weekly active days) and guardrails (e.g., session length, retention). Then outline a randomized experiment (e.g., encourage social feature usage) with proper power analysis, and if randomization isn't feasible, propose an observational study using propensity score matching or instrumental variables. Address multiple comparisons, validity threats, and end with a decision rule and communication plan.

Pro tip: Emphasize that engagement metrics should be tied to long-term user value, not just short-term activity, and always include guardrails to catch negative side effects. Also, when computing sample size, show the formula and plug in numbers to demonstrate rigor.

1. Define Metrics and Guardrails

Choose a primary engagement metric (e.g., weekly active days per user) that aligns with business goals, and set guardrails (e.g., session length, 7-day retention) to monitor for unintended harm.

2. Design Randomized Experiment

Propose an A/B test where users are randomized to receive social feature prompts vs. game feature prompts, ensuring proper randomization and blinding. Compute sample size using the two-proportion z-test formula with given parameters.

3. Observational Fallback with Causal Inference

If randomization is not possible, use propensity score matching or instrumental variables to estimate the causal effect of social feature usage on engagement, adjusting for confounders.

4. Address Multiple Comparisons and Validity Threats

Apply corrections like Bonferroni or Benjamini-Hochberg for multiple metrics, and list at least five threats (e.g., selection bias, novelty effect) with robustness checks (e.g., sensitivity analysis, placebo tests).

5. Decision Rule and Communication Plan

Define a clear decision rule (e.g., launch if primary metric improves by ≥3pp with p<0.05 and guardrails not violated) and outline how to communicate results to stakeholders, including caveats and next steps.

Key Points to Mention

Primary engagement metric: weekly active days per user; guardrails: session length, 7-day retention, crash rate.
Sample size calculation: n = (Zα/2 + Zβ)^2 * (p1(1-p1) + p2(1-p2)) / (p1-p2)^2, with p1=0.35, p2=0.38, yielding ~3,000 per group.
Randomized experiment: randomize at user level, use intent-to-treat analysis, and consider cluster randomization if social features have network effects.
Observational methods: propensity score matching, difference-in-differences, or instrumental variables (e.g., feature rollout timing).
Multiple comparisons: control false discovery rate using Benjamini-Hochberg; pre-register primary and secondary metrics.
Validity threats: selection bias, confounding, novelty effect, spillover, measurement error; robustness checks: sensitivity analysis, negative control outcomes, subgroup analyses.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.