Airbnb·Data Scientist·Technical Phone Screen·Senior

Senior

May 2025Remote

Summary

Airbnb data science interview with a brutally hard causal inference question about a global launch with no holdout. One question, no warmup, just straight into the deep end.

Questions Asked (1)

A product feature was shipped globally on a single date with no control group or holdout. How would you design a plan to estimate its causal effect on weekly revenue and 7-day retention? Walk through at least two independent identification strategies, covering assumptions, data needs, diagnostics, uncertainty quantification, how you'd bound effects if assumptions break, how you'd handle confounders like seasonality or concurrent campaigns, and how you'd ultimately reconcile conflicting estimates into a ship/rollback/iterate recommendation.

A/B Testing & ExperimentationProduct Analytics & MetricsTechnical Trade-offs

Author's notes

This wrecked me a little.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Acknowledge the lack of a control group and propose at least two independent quasi-experimental designs, such as synthetic control and difference-in-differences, to triangulate the causal effect. For each, detail assumptions, data requirements, diagnostics, and sensitivity analyses, then reconcile estimates into a clear ship/rollback/iterate recommendation.

Pro tip: Emphasize that no single method is definitive; triangulating multiple designs with different assumptions strengthens causal claims and helps quantify uncertainty. Also, proactively discuss how you'd bound effects if key assumptions fail, showing robustness.

1. Clarify the causal question and data landscape

Define the estimand (ATT of global launch on weekly revenue and 7-day retention) and inventory available data: pre/post periods, potential control units (e.g., similar markets, user segments), covariates, and known confounders like seasonality and concurrent campaigns.

2. Design two independent identification strategies

Propose e.g., (1) Synthetic Control Method (SCM) using pre-launch data to construct a weighted combination of untreated markets, and (2) Difference-in-Differences (DiD) with a carefully chosen comparison group (e.g., markets with delayed rollout or similar pre-trends). For each, specify assumptions (e.g., parallel trends, no spillovers), data needs, and diagnostics (pre-trend tests, placebo tests).

3. Quantify uncertainty and bound effects

Use placebo tests, bootstrapping, or Bayesian methods to quantify uncertainty. If assumptions are questionable, derive bounds (e.g., Rosenbaum bounds for unmeasured confounding, or partial identification) to show the range of plausible effects.

4. Address confounders and reconcile estimates

Adjust for seasonality (e.g., time fixed effects, seasonal decomposition) and concurrent campaigns (e.g., include campaign indicators, use propensity score weighting). Compare estimates from both strategies; if they conflict, investigate sources (e.g., different assumptions, data issues) and consider a meta-analytic or Bayesian model averaging approach.

5. Make a ship/rollback/iterate recommendation

Synthesize findings: if both methods show a positive, robust effect, recommend ship; if negative or highly uncertain, recommend rollback or iterate. Clearly state the decision criteria (e.g., effect size, confidence intervals, business impact) and acknowledge limitations.

Key Points to Mention

Synthetic Control Method (SCM) and its assumptions (e.g., convex hull, no interference)
Difference-in-Differences (DiD) with parallel trends assumption and pre-trend testing
Placebo tests and permutation inference for uncertainty quantification
Sensitivity analysis (e.g., Rosenbaum bounds) to assess robustness to unmeasured confounding
Handling seasonality via time fixed effects or seasonal adjustment, and concurrent campaigns via covariate adjustment
Triangulation and reconciliation of estimates from multiple designs to inform decision-making

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.