Meta·Data Scientist·Technical Phone Screen·Intermediate

Intermediate

May 2026

Summary

Meta data scientist interview with a product analytics angle. The main question was about building a fake-user classifier and picking the right evaluation metrics, which sounds straightforward but turns into a whole conversation about business tradeoffs pretty fast.

Questions Asked (1)

You're seeing a spike in daily average comments that might be driven by fake users. You need to build a classifier to detect them. What evaluation metrics would you use and why?

Product Analytics & MetricsTechnical Trade-offsRoot Cause Analysis

Author's notes

I started with F1 and ROC-AUC because those felt safe, but the follow-up pushed me on the actual business cost side and that's where I got a bit wobbly.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by framing the problem as an imbalanced classification task where false positives (flagging real users) and false negatives (missing fake users) have different business costs. Then recommend metrics that align with the goal of detecting fake users while minimizing harm to genuine user experience, such as precision, recall, F1, and PR-AUC, and justify why accuracy is misleading.

Pro tip: Mention that you would also monitor the impact on the overall comment metric and consider a cost-sensitive evaluation, since at Meta the ultimate goal is to improve user experience, not just model performance.

1. Clarify the business objective and costs

Understand why detecting fake users matters: to maintain trust and reduce spam. Discuss the relative costs of false positives (blocking real users) and false negatives (allowing fake users).

2. Identify the class imbalance

Acknowledge that fake users are likely a small minority, making accuracy a poor metric. Emphasize the need for metrics robust to imbalance.

3. Select primary evaluation metrics

Recommend precision, recall, F1-score, and PR-AUC. Explain that precision measures how many flagged users are truly fake, while recall measures how many fake users are caught.

4. Consider secondary and business metrics

Suggest additional metrics like ROC-AUC for ranking, and business metrics such as reduction in fake comments or impact on engagement. Mention cost-sensitive evaluation if costs are known.

5. Validate and iterate

Propose using a holdout set, cross-validation, and monitoring model performance over time. Discuss threshold tuning based on business needs.

Key Points to Mention

Class imbalance makes accuracy misleading; use precision, recall, F1, and PR-AUC.
False positives (flagging real users) can harm user trust and engagement; false negatives allow spam.
PR-AUC is preferred over ROC-AUC for highly imbalanced datasets.
Cost-sensitive evaluation: assign costs to FP and FN and optimize accordingly.
Business impact metrics: reduction in fake comments, user reports, and overall engagement.
Threshold selection should align with business goals (e.g., high precision if false positives are costly).

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.