Netflix·Data Scientist·Technical Phone Screen·Senior

Senior

Jun 2026

Summary

Netflix data scientist interview with a meaty system design question around fraud detection. Single question but it covered a lot of ground, from feature engineering all the way to post-deployment monitoring.

Questions Asked (1)

Walk through the full pipeline for training and deploying an XGBoost model to detect fraudulent subscription payments in real time. What evaluation metrics matter most, how do you deal with heavy class imbalance, and how would you catch model drift after the thing is live?

System DesignTechnical Trade-offsProduct Analytics & Metrics

Author's notes

This question is basically four questions stitched together and I felt it the whole time.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Structure your answer as an end-to-end system design: start with data ingestion and feature engineering, then cover training with imbalance handling, evaluation metrics, deployment for real-time scoring, and finally monitoring for drift. Emphasize trade-offs between precision and recall, latency constraints, and how you'd iterate post-deployment.

Pro tip: Netflix cares about member experience, so frame false positives as a major cost (blocking legitimate payments) and propose a human-in-the-loop or step-up authentication for borderline cases. Also, mention that you'd log all predictions and outcomes to enable continuous evaluation and retraining.

1. Data & Feature Pipeline

Describe how you'd collect and preprocess data: historical transactions, user behavior, device info, etc. Highlight real-time feature computation (e.g., using Flink or Kafka Streams) and point-in-time correctness to avoid leakage.

2. Model Training & Imbalance Handling

Explain training XGBoost with techniques like scale_pos_weight, SMOTE, or undersampling. Discuss hyperparameter tuning and cross-validation with stratified folds.

3. Evaluation Metrics

Choose metrics beyond accuracy: PR-AUC, recall at fixed precision, F1, and cost-based metrics. Emphasize that in fraud detection, recall is often prioritized but must balance with false positives.

4. Deployment & Real-time Scoring

Outline serving the model via a low-latency API (e.g., using ONNX or Treelite), with feature store integration. Discuss A/B testing and shadow mode before full rollout.

5. Monitoring & Drift Detection

Describe tracking prediction distributions, feature drift (e.g., PSI, KL divergence), and performance metrics over time. Set up alerts and a retraining pipeline triggered by drift or performance degradation.

Key Points to Mention

Class imbalance techniques: scale_pos_weight, SMOTE, and evaluation with PR-AUC instead of ROC-AUC.
Real-time feature engineering with point-in-time correctness to prevent data leakage.
Latency constraints for real-time scoring and model serialization formats (ONNX, Treelite).
Cost-sensitive evaluation: false positives (blocking legitimate users) vs false negatives (fraud loss).
Drift detection methods: population stability index (PSI), KL divergence, and monitoring prediction distributions.
Continuous retraining pipeline and human-in-the-loop for borderline cases.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.