Meta·Data Scientist·Technical Phone Screen·Senior

Senior

Jun 2026

Summary

Meta DS interview with a meaty A/B testing case that had a twist built right into the setup. The question looked like a standard experiment evaluation until the segmentation numbers came out and things got complicated fast.

Questions Asked (1)

An A/B test for a new ad-ranking algorithm shows a 5% overall CTR lift but a 100% lift specifically for Indian males aged 18-24. Walk through what analyses you'd run to decide whether to launch, list possible root causes for that heterogeneous effect, and describe what additional data or follow-up experiments you'd need before rollout.

A/B Testing & ExperimentationRoot Cause AnalysisProduct Analytics & Metrics

Author's notes

This one took me a second to even figure out where to start.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by validating the overall 5% lift and the segment-specific 100% lift through rigorous statistical checks, then investigate potential root causes for the heterogeneity, and finally outline a plan for additional data collection and follow-up experiments to de-risk the launch decision.

Pro tip: Emphasize the importance of checking for novelty effects and segment-specific biases, and propose a holdout experiment to measure long-term impact, showing you think beyond immediate metrics.

1. Validate the results

Check statistical significance, confidence intervals, and power for both overall and segment-level lifts. Ensure the segment lift isn't due to multiple testing or small sample size.

2. Investigate root causes

Explore possible reasons for the heterogeneous effect, such as data quality issues, novelty effects, segment-specific user behavior, or algorithmic bias.

3. Assess business impact and risks

Evaluate whether the overall lift is driven by the segment and if it's sustainable. Consider potential negative impacts on other segments or long-term user experience.

4. Plan follow-up experiments

Design targeted experiments to confirm the effect, test generalizability, and measure long-term metrics. Include holdout groups and segment-specific analyses.

5. Make a launch decision

Synthesize findings to recommend launch, iterate, or abandon, with conditions for monitoring and rollback.

Key Points to Mention

Statistical significance and multiple comparisons correction (e.g., Bonferroni, FDR)
Novelty effect and primacy effect
Segment definition and potential data leakage or misclassification
Simpson's paradox and confounding variables
Long-term metrics and holdout experiments
Algorithmic fairness and bias detection

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.