Meta·Data Scientist·Onsite - System Design / Architecture·Senior

SeniorPrefer not to say

Jun 2026

Summary

Brutal Meta DS onsite question about staging a feature rollout using only pre-launch data. The whole thing was one long end-to-end ML design problem and it went in a lot of directions fast.

Questions Asked (1)

You need to predict which users or products would benefit most from a new 'More like this' feature before it launches globally. Using only pre-launch interaction and product data plus any logging you can add at launch, design an end-to-end modeling approach covering: proxy labeling and post-launch true labels (with target leakage prevention), feature engineering including cross-features and cold-start handling, two candidate model families with calibration considerations, offline evaluation strategy and shadow testing, and a deployment policy that addresses fairness, negative spillovers, and ongoing drift monitoring.

System DesignA/B Testing & ExperimentationProduct Analytics & Metrics

Author's notes

This was basically five questions crammed into one.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Structure your answer around the ML lifecycle: start with proxy label design and leakage-safe feature engineering, then compare two model families with calibration and cold-start strategies, and finish with offline evaluation, shadow testing, and a deployment policy that includes fairness, spillover mitigation, and drift monitoring. Emphasize how you would validate proxies and iterate using launch logging.

Pro tip: Propose a 'proxy validation' step: before modeling, correlate your proxy labels with a small human-labeled set or a holdout experiment to ensure they are predictive of true engagement. This shows rigor and prevents building on a flawed target.

1. Define proxy labels and plan for true labels

Use pre-launch interaction data (e.g., co-clicks, co-views, co-purchases) to create proxy labels for 'benefit'. At launch, log true labels (e.g., clicks on 'More like this' recommendations) and ensure no leakage by using only pre-launch features for training.

2. Engineer features with cross-features and cold-start handling

Create user, product, and interaction features; include cross-features (e.g., user-category affinity). For cold-start, use content-based features, metadata, and global popularity; consider a separate model or fallback for new entities.

3. Select and calibrate two model families

Choose e.g., matrix factorization (collaborative filtering) and gradient boosted trees (with cross-features). Calibrate outputs (e.g., Platt scaling) to ensure predicted probabilities align with true engagement rates, especially for ranking.

4. Offline evaluation and shadow testing

Use time-based splits and evaluate with ranking metrics (NDCG, MAP) and calibration plots. Run shadow testing (log model predictions without affecting users) to compare against production and detect issues before full launch.

5. Deployment policy: fairness, spillovers, and drift monitoring

Define a policy that ensures fair exposure across user segments, mitigates negative spillovers (e.g., cannibalization), and includes ongoing drift monitoring (e.g., PSI, retraining triggers). Use A/B testing to measure true impact.

Key Points to Mention

Proxy label validation: correlate proxies with true labels or human judgments to ensure they capture the intended benefit.
Target leakage prevention: strictly separate pre-launch features from post-launch labels; use time-based splits.
Cold-start strategies: content-based features, metadata, and fallback models for new users/products.
Calibration: apply Platt scaling or isotonic regression to align predicted probabilities with true rates.
Shadow testing: deploy model in shadow mode to compare predictions with production without affecting users.
Fairness and spillovers: monitor exposure across segments, test for cannibalization, and set guardrail metrics.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.