Meta·Data Scientist·Technical Phone Screen·Senior

SeniorPrefer not to say

Jun 2026Remote

Summary

A technical screen for a Data Scientist role at Meta focused entirely on one dense experimentation question set inside WhatsApp's Calling org. The whole thing was a single scenario about designing an A/B test for an adaptive codec feature, but it had about eight sub-problems packed into it. Left feeling like I answered the broad strokes but fumbled some of the math under pressure.

Questions Asked (1)

You're joining WhatsApp's Calling team. A PM wants to test an adaptive codec that should reduce call drops on unstable networks. Design a full A/B test: choose and justify your randomization unit given two-sided exposure and interference, define your primary and guardrail metrics precisely, specify the exposure logic (does the feature activate only when both sides are treated or just the caller?), outline a ramp plan with spillover checks, handle non-independence from repeat callers and seasonality, do a power calculation back-of-envelope given a 3.2% baseline drop rate and 8% relative reduction target, compute the design effect using a caller-level ICC of 0.10, estimate unique callers per arm needed, and state your estimand and estimator including how you'd diagnose interference across treatment arms.

A/B Testing & ExperimentationProduct Analytics & Metrics

Author's notes

This was basically a take-home crammed into a live call.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by framing the experiment around the two-sided nature of calls: choose the caller as the randomization unit but acknowledge interference and plan to measure it. Define metrics and exposure logic clearly, then walk through power analysis with design effect adjustment, and finally discuss estimand and diagnostics for interference.

Pro tip: Always pre-specify how you'll handle partial exposure (e.g., one side treated) and consider using a cluster-randomized design or switchback if interference is severe; this shows you understand real-world constraints beyond textbook A/B tests.

1. Randomization Unit and Justification

Choose the caller as the randomization unit because calls are initiated by one user, but note that two-sided exposure creates interference. Justify that caller-level randomization balances user experience and allows measuring spillover via call graphs.

2. Metrics and Exposure Logic

Define primary metric as call drop rate per call, and guardrail metrics like call setup success rate, audio quality (MOS), and user retention. Specify exposure: feature activates only when both caller and callee are treated to avoid asymmetric effects, but also track partial exposure cases.

3. Ramp Plan and Spillover Checks

Outline a phased ramp: start with 1% of callers, monitor for interference via network effects (e.g., call success rate of untreated partners of treated callers). Use geo or time-based holdouts to detect spillover.

4. Power Calculation and Design Effect

Back-of-envelope: baseline drop rate 3.2%, target 8% relative reduction (to 2.944%). Compute sample size for independent units, then adjust for design effect (1 + (m-1)*ICC) with ICC=0.10 and average calls per caller m. Estimate unique callers per arm.

5. Estimand, Estimator, and Interference Diagnostics

State estimand as average treatment effect on call drop rate among treated callers. Use intention-to-treat (ITT) estimator with cluster-robust standard errors. Diagnose interference by comparing outcomes for treated vs. untreated partners, and use causal graph or network analysis.

Key Points to Mention

Two-sided exposure and interference: randomization at caller level but calls involve two users, so treatment of one side can affect the other.
Design effect formula: 1 + (m-1)*ICC, where m is average calls per caller; with ICC=0.10, need to inflate sample size.
Power calculation: baseline 3.2%, MDE 8% relative (0.256 pp absolute), alpha=0.05, power=0.80; compute n per arm then adjust for design effect.
Exposure logic: feature activates only when both sides treated; track partial exposure as a separate analysis.
Ramp plan: phased rollout with spillover checks using network clusters or geo holdouts.
Estimand: average treatment effect on the treated (ATT) or ITT; estimator: difference in means with cluster-robust SEs; diagnostics: compare outcomes for partners of treated vs. control.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.