Meta·Data Scientist·Technical Phone Screen·Intermediate

Intermediate

Jun 2026

Summary

Meta data scientist screen with a classic Bayesian probability question dressed up in a security context. Pretty short and focused, just the one problem but it requires you to actually work through the math on the spot.

Questions Asked (1)

A security model has 95% accuracy on both good and bad users. Only 5% of users are actually bad. If the model flags someone as bad, what's the probability they really are bad?

Product Analytics & MetricsRoot Cause Analysis

Author's notes

The number that comes out is way lower than you'd expect and I remember second-guessing my arithmetic twice.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Recognize this as a Bayes' theorem problem and set up the calculation using the given base rate and accuracy. Compute the probability that a flagged user is actually bad by considering both true positives and false positives. Clearly state the assumptions and interpret the result in the context of the model's practical utility.

Pro tip: Emphasize that high accuracy can be misleading with imbalanced classes, and discuss how the base rate (5% bad) drastically reduces the precision of a positive flag. This shows you understand the difference between accuracy and precision, a key concept in applied data science.

1. Identify the given probabilities

Extract the base rate (P(Bad) = 0.05), sensitivity (P(Flag|Bad) = 0.95), and specificity (P(Not Flag|Good) = 0.95). Note that P(Good) = 0.95.

2. Apply Bayes' theorem

Use the formula P(Bad|Flag) = P(Flag|Bad) * P(Bad) / [P(Flag|Bad)*P(Bad) + P(Flag|Good)*P(Good)].

3. Compute the numerator and denominator

Calculate the numerator: 0.95 * 0.05 = 0.0475. Calculate the denominator: 0.0475 + (0.05 * 0.95) = 0.0475 + 0.0475 = 0.095.

4. Calculate the final probability

Divide the numerator by the denominator: 0.0475 / 0.095 = 0.5. So the probability is 50%.

5. Interpret the result

Explain that despite 95% accuracy, only half of flagged users are actually bad due to the low base rate. Discuss implications for precision and potential improvements.

Key Points to Mention

Bayes' theorem and conditional probability
Base rate fallacy and class imbalance
Difference between accuracy and precision
Confusion matrix concepts (true positives, false positives)
Practical implications for security models (e.g., cost of false positives)
Potential need for additional features or threshold tuning to improve precision

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.