Meta·Data Scientist·Technical Phone Screen·Intermediate

Intermediate

Jun 2026

Summary

Meta data scientist interview, one technical round focused entirely on model evaluation for a fake account detection system. Pretty conceptual but they pushed hard on the business side of things, not just the stats.

Questions Asked (1)

You've built a classifier to detect fake accounts and leadership wants proof it's working in production. Which evaluation metrics would you use and why? Walk through the trade-offs between precision, recall, F1, and ROC-AUC, and factor in the business costs of false positives versus false negatives.

Product Analytics & MetricsTechnical Trade-offsA/B Testing & Experimentation

Author's notes

I started with the usual precision vs recall framing and talked about class imbalance since fake accounts are a small fraction of all accounts.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by framing the problem as a business decision, not just a modeling exercise: define what a false positive and false negative cost Meta in terms of user experience and revenue. Then recommend a primary metric (likely precision at a fixed recall or PR-AUC) and a monitoring plan, while explaining why accuracy and ROC-AUC can be misleading for imbalanced fake-account detection. Finally, tie the metric choice to a production experiment or holdout validation to prove impact.

Pro tip: Anchor your answer in Meta's scale and adversarial context: fake accounts are a tiny fraction of traffic and attackers adapt, so you should emphasize precision at high recall, PR-AUC, and continuous monitoring with drift detection rather than a one-time offline metric.

1. Clarify business costs and constraints

Quantify the cost of false positives (e.g., blocking real users, support load, revenue loss) versus false negatives (e.g., spam, fraud, policy violations, user trust erosion). Establish the acceptable operating point based on these costs.

2. Choose primary and secondary metrics

Select precision at a fixed high recall (or recall at a fixed precision) as the primary metric because it directly reflects the business trade-off. Use PR-AUC as a threshold-independent summary and F1 as a balanced but often less business-aligned secondary metric.

3. Explain trade-offs and why not ROC-AUC/accuracy

Discuss how precision and recall trade off via threshold; F1 assumes equal cost and may hide asymmetry; ROC-AUC can be overly optimistic under extreme class imbalance because it incorporates true negatives. Emphasize that accuracy is meaningless when positives are rare.

4. Propose production validation and monitoring

Recommend an A/B test or holdout with guardrail metrics (e.g., user reports, account creation success) to prove the model works in production. Set up continuous monitoring of precision/recall, data drift, and adversarial adaptation.

5. Summarize recommendation and next steps

Conclude with a clear recommendation: use precision at 90% recall as the north-star metric, track PR-AUC and F1 for model comparison, and run a production experiment to measure business impact. Suggest periodic re-evaluation as attacker behavior evolves.

Key Points to Mention

Class imbalance: fake accounts are a small fraction of traffic, making accuracy and ROC-AUC misleading; PR-AUC is more informative.
Business cost asymmetry: false positives (blocking real users) often cost more than false negatives (missing some fake accounts) at Meta's scale.
Precision-recall trade-off: lowering threshold increases recall but decreases precision; choose operating point based on cost ratio.
F1 assumes equal weight for precision and recall, which may not align with business priorities; use only if costs are balanced.
ROC-AUC can be optimistic under imbalance because it includes true negatives, which are abundant; PR-AUC focuses on the positive class.
Production proof: use A/B test or holdout with guardrail metrics (e.g., user reports, engagement) and monitor for drift and adversarial adaptation.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.