Airbnb·Machine Learning Engineer·Onsite - System Design / Architecture·Senior

SeniorPrefer not to say

Jul 2026

Summary

Airbnb ML system design round, one big question covering the full recommendation pipeline. Pretty dense for a single session but the scope was clear from the start.

Questions Asked (1)

Design a large-scale recommendation system end to end, covering candidate generation, ranking, feature engineering, model serving, online vs offline metrics, A/B testing, and how changes to a downstream scoring service can affect live metrics.

System DesignA/B Testing & ExperimentationTechnical Trade-offs

Author's notes

This is basically a full ML systems interview crammed into one question.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by clarifying requirements and scale, then walk through the end-to-end pipeline from candidate generation to serving, emphasizing trade-offs at each stage. Integrate experimentation and monitoring throughout, and explicitly discuss how downstream changes can impact online metrics.

Pro tip: Anchor your design in Airbnb's two-sided marketplace by discussing how to balance guest and host objectives, and use concrete examples like similar listing embeddings for candidate generation. Also, emphasize that offline metrics are proxies—always validate with online A/B tests and be wary of feedback loops.

1. Clarify Requirements and Scale

Ask about scale (users, listings, QPS), latency budgets, business objectives (e.g., bookings, guest satisfaction), and constraints (e.g., real-time features). This sets the stage for design decisions.

2. Design the Pipeline: Candidate Generation to Ranking

Describe candidate generation (e.g., collaborative filtering, embeddings, geographic retrieval) and ranking (e.g., GBDT, DNN) with feature engineering (user, listing, context, cross features). Discuss trade-offs between model complexity and latency.

3. Model Serving and Infrastructure

Explain serving architecture: offline training, online inference (e.g., TF Serving, feature store), caching, and fallbacks. Address scalability, latency, and reliability.

4. Metrics and Experimentation

Define offline metrics (e.g., NDCG, recall@k) and online metrics (e.g., CTR, bookings, revenue). Describe A/B testing setup, guardrail metrics, and how to detect downstream impacts (e.g., scoring service changes causing metric shifts).

5. Monitor, Iterate, and Handle Downstream Effects

Discuss monitoring, alerting, and debugging. Explain how changes in a downstream service (e.g., pricing, availability) can affect live metrics and how to isolate causes via experiments and causal inference.

Key Points to Mention

Two-stage architecture: candidate generation (high recall) and ranking (high precision) to balance scalability and accuracy.
Feature engineering: user features (history, demographics), listing features (amenities, location), context (time, device), and cross features (user-listing interactions).
Online vs offline metrics: offline metrics like NDCG for ranking quality, online metrics like CTR and bookings; beware of offline-online mismatch.
A/B testing: randomization unit, sample size, guardrail metrics, and sequential testing to avoid peeking.
Downstream service impact: changes in scoring service (e.g., price ranking) can alter user behavior and cascade to other metrics; use holdbacks and causal analysis.
Serving considerations: low-latency inference, feature freshness, model versioning, and fallback strategies.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.