Airbnb·Data Scientist·Technical Phone Screen·Senior

Senior

Jun 2026

Summary

Airbnb data scientist interview that dropped me into a product-planning scenario where I had to define the full experiment framework before a single line of code gets written. Pretty rigorous, covers a lot of ground in one question.

Questions Asked (1)

You're in a product planning session before development starts. Walk through the metrics framework, guardrails, experiment design, unit of randomization, sample size calculation, and how you'd monitor the experiment while it's running.

A/B Testing & ExperimentationProduct Analytics & MetricsProduct Strategy

Author's notes

This is a lot to hold in your head at once and I think I fumbled the ordering.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by framing the experiment around a clear product hypothesis and the decision it will inform, then walk through the metrics hierarchy (primary, secondary, guardrails), randomization unit, power analysis, and monitoring plan in a logical sequence. Emphasize trade-offs and practical constraints at Airbnb's scale, such as network effects and seasonality.

Pro tip: Show you understand that experiment design is not just statistical but also operational: mention how you'd pre-register the analysis plan and set up automated alerts for guardrail metrics to catch issues early without p-hacking.

1. Define hypothesis and decision

Articulate the product change, the expected impact, and the decision the experiment will inform (e.g., launch, iterate, or kill). This anchors the metrics and design choices.

2. Choose metrics and guardrails

Select a primary success metric tied to the hypothesis, secondary metrics for deeper insight, and guardrail metrics to ensure no harm to user experience or business health.

3. Design experiment: randomization unit and sample size

Determine the randomization unit (e.g., user, listing, city) based on interference risks, then calculate required sample size using power analysis, accounting for baseline rates and minimum detectable effect.

4. Plan monitoring and analysis

Set up real-time dashboards for key metrics, define stopping rules, and pre-register the analysis plan including segmentation and multiple testing corrections.

Key Points to Mention

Primary metric should be directly tied to the hypothesis and sensitive to the change; guardrails include latency, revenue, and user satisfaction.
Randomization unit: user-level for most cases, but consider cluster randomization (e.g., city) if network effects or marketplace dynamics exist.
Sample size calculation: use power analysis (e.g., 80% power, 5% significance) with estimated baseline and MDE; adjust for multiple comparisons.
Monitoring: track sample ratio mismatch (SRM), guardrail metrics, and early signals; avoid peeking without correction.
Consider novelty effects, seasonality, and long-term holdout groups for sustained impact.
Pre-register the analysis plan to prevent p-hacking and ensure valid inference.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.