Meta·Data Scientist·Technical Phone Screen·Senior

Senior

Apr 2026

Summary

Brutal Meta DS interview centered on a single massive experiment design question. The kind of prompt where you think you understand it and then realize twenty minutes in you've only scratched the surface.

Questions Asked (1)

You have 1,200 EU businesses in an early-access pool with heavy-tailed chat volumes and expected 20-30% initial subscription take-up. Design a launch test to estimate impact on subscription revenue per business and Resolved-Within-24h Rate. Cover your randomization unit, stratification approach, sample size and power under overdispersion, interference risks from shared customers, ITT vs TOT handling, sequential monitoring, and a pre-registered analysis plan with a decision rule to roll out, iterate, or stop.

A/B Testing & ExperimentationProduct Analytics & MetricsTechnical Trade-offs

Author's notes

This is basically a full experiment design dissertation compressed into one interview question.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by framing the experiment as a cluster-randomized trial at the business level to handle interference from shared customers, then detail stratification by volume and take-up propensity. Explain how you would size the experiment accounting for overdispersion, monitor sequentially with alpha-spending, and pre-register an analysis plan that includes ITT and TOT estimates with a clear decision rule.

Pro tip: Emphasize that with heavy-tailed volumes, you should use a log transformation or a robust variance estimator for revenue per business, and consider a CUPED adjustment using pre-experiment data to increase power. Also, pre-register a non-inferiority margin for Resolved-Within-24h Rate to ensure you don't degrade support quality while chasing revenue.

1. Define randomization and stratification

Randomize at the business level to avoid contamination from shared customers, and stratify by historical chat volume (e.g., top 1%, 1-10%, rest) and predicted take-up propensity to balance heavy tails and improve power.

2. Determine sample size and power under overdispersion

Use simulation or negative binomial models to account for overdispersion in chat volumes; calculate required sample size to detect a minimum detectable effect (e.g., 5% lift in revenue per business) with 80% power and 5% significance, considering the 20-30% take-up.

3. Address interference and treatment effects

Acknowledge interference risks from shared customers across businesses; use cluster randomization and consider a design that isolates shared customers. Plan to estimate both ITT (effect of assignment) and TOT (effect of subscription) using instrumental variables or compliance-adjusted analysis.

4. Implement sequential monitoring and decision rules

Use group sequential testing with alpha-spending (e.g., O'Brien-Fleming) to allow early stopping for efficacy or futility, and pre-register a decision rule: roll out if revenue lift ≥ X and Resolved-Within-24h Rate non-inferior; iterate if revenue lift positive but quality degraded; stop if no lift or harm.

5. Pre-register analysis plan

Document primary and secondary metrics, handling of outliers, transformations, covariates (e.g., CUPED), and multiple testing corrections. Specify the model (e.g., mixed-effects or GEE) and sensitivity analyses to ensure robustness.

Key Points to Mention

Cluster randomization at business level to mitigate interference from shared customers
Stratification by chat volume and take-up propensity to handle heavy tails
Sample size calculation using negative binomial or simulation to account for overdispersion
Sequential monitoring with alpha-spending and pre-registered stopping rules
ITT vs TOT estimation using instrumental variables or compliance-adjusted analysis
Pre-registered analysis plan with decision rule to roll out, iterate, or stop

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.