Meta·Data Scientist·Technical Phone Screen·Senior

Senior

May 2026

Summary

Meta data scientist interview with a meaty stats/probability question that combined Bayes' theorem, confusion matrices, and real arithmetic. The kind of question that sounds manageable until you're actually doing the math live.

Questions Asked (1)

You have a classifier that flags potentially bad actors on a platform. Given a prior probability p that any user is a bad actor, and known true-positive and false-positive rates for the classifier, derive the posterior probability that a flagged user is actually bad. Then define Type I and Type II errors in this context, explain the business tradeoffs each represents, and work through a concrete example: 1% of 10 million users are bad actors, the classifier has 95% recall and a 2% false-positive rate. How many bad actors get caught, and how many good users get wrongly flagged?

A/B Testing & ExperimentationProduct Analytics & MetricsAlgorithms & Data Structures

Author's notes

I started okay with the Bayes derivation but fumbled the arithmetic partway through.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by stating Bayes' theorem for the posterior probability, then define Type I and Type II errors with their business implications. Finally, compute the concrete example step-by-step, emphasizing the base rate fallacy and the high false positive count relative to true positives.

Pro tip: Always highlight the base rate fallacy: with a low prior, even a highly accurate classifier can yield many false positives, so precision may be low. Discuss how adjusting the threshold or using additional signals can mitigate this in practice.

1. Derive Posterior Probability

Use Bayes' theorem: P(Bad | Flagged) = (TPR * p) / (TPR * p + FPR * (1-p)). Clearly define TPR (recall) and FPR.

2. Define Type I and Type II Errors

Type I error: false positive (flagging a good user). Type II error: false negative (missing a bad actor). Explain each in context.

3. Explain Business Tradeoffs

Type I errors lead to user frustration, support costs, and unfair penalties. Type II errors allow bad actors to remain, risking platform integrity and user trust. Discuss the balance based on business priorities.

4. Compute Concrete Example

Calculate: 10M users, 1% bad = 100,000 bad actors. Recall 95% => 95,000 caught. FPR 2% of 9.9M good = 198,000 false positives. Total flagged = 293,000. Precision = 95,000/293,000 ≈ 32.4%.

5. Interpret and Discuss Implications

Highlight that despite high recall, precision is low due to many false positives. Discuss potential improvements like threshold tuning, additional features, or manual review.

Key Points to Mention

Bayes' theorem formula for posterior probability
Type I error = false positive, Type II error = false negative
Business impact: false positives harm user experience and trust; false negatives allow bad actors to persist
Base rate fallacy: low prevalence leads to low precision even with high accuracy
Concrete numbers: 95,000 true positives, 198,000 false positives, precision ~32.4%
Strategies to mitigate: adjust threshold, ensemble models, human-in-the-loop

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.