Meta·Data Scientist·Technical Phone Screen·Senior

Senior

Jun 2026

Summary

Meta DS interview with a causal inference question that felt more like a research design problem than anything I'd call typical analytics. The setup was straightforward but the depth they expected was not.

Questions Asked (1)

Management believes revenue growth from a particular traffic source is coming at the expense of other sources rather than being genuinely new growth. How would you design an analysis to test whether next year's revenue increase from that source is offset by declines elsewhere?

A/B Testing & ExperimentationProduct Analytics & MetricsRoot Cause Analysis

Author's notes

This one tripped me up because my first instinct was to just plot the trends side by side and call it a day.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by clarifying the business question and defining the metric of interest, then propose a rigorous causal inference framework (e.g., geo-based holdout or synthetic control) to isolate the incremental effect of the traffic source. Compare the observed revenue lift in the source against changes in other channels, using a difference-in-differences or cannibalization analysis to test for offsetting declines.

Pro tip: Emphasize the importance of pre-registering the analysis plan and checking for pre-existing trends to avoid false positives; also, consider that cannibalization may be partial and time-lagged, so analyze both short-term and long-term effects.

1. Define the hypothesis and metrics

Clearly state the null hypothesis (no cannibalization) and alternative (cannibalization exists). Define primary metrics: revenue from the target source and total revenue, plus secondary metrics like revenue from other sources and user-level conversion rates.

2. Choose a causal inference design

Select an appropriate experimental or quasi-experimental design, such as a geo-based holdout (randomly assign regions to keep or remove the traffic source), a switchback experiment, or synthetic control if randomization is not feasible.

3. Analyze incremental revenue and cannibalization

Estimate the incremental revenue from the target source using the chosen design. Then, test for cannibalization by comparing changes in other sources between treatment and control groups, using difference-in-differences or a cannibalization rate metric.

4. Check robustness and heterogeneity

Validate results with placebo tests, sensitivity analyses, and segment-level breakdowns (e.g., by user tenure, device, region) to see if cannibalization varies across subgroups.

5. Communicate findings and recommendations

Summarize whether the revenue increase is incremental or offset, quantify the net impact, and provide actionable recommendations for budget allocation or further testing.

Key Points to Mention

Cannibalization vs. incrementality: distinguish between total revenue growth and net new revenue.
Experimental design: geo-based holdout, switchback, or randomized controlled trial (RCT) to establish causality.
Difference-in-differences (DiD) and synthetic control methods for quasi-experimental settings.
Cannibalization rate calculation: (decline in other sources) / (increase in target source).
Time-lagged effects and long-term holdout to capture delayed cannibalization.
Segmentation analysis to identify which channels or user groups are most affected.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.