Meta·Data Scientist·Technical Phone Screen·Intermediate

Intermediate

May 2026

Summary

Meta data science technical screen with a probability/Bayesian reasoning question that looked straightforward on the surface but had a real gotcha in part (c).

Questions Asked (1)

In a population where 5% of users are 'bad' and 95% are 'good', a model has 95% sensitivity and 95% specificity. If the model flags someone as 'bad', what's the actual probability they're bad? And if it predicts 'good', what's the probability they really are good? Then explain how those probabilities shift as prevalence changes and what that tells us about base rates.

Product Analytics & MetricsA/B Testing & Experimentation

Author's notes

The math for parts (a) and (b) is just Bayes' theorem.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

First, compute the confusion matrix using the given prevalence, sensitivity, and specificity. Then calculate the positive predictive value (PPV) and negative predictive value (NPV). Finally, discuss how these metrics change with prevalence and the implications for base rates.

Pro tip: Always relate the result to the business context: a low PPV means many false positives, which can be costly or annoying to users, so consider precision-recall trade-offs.

1. Define the given metrics

Clarify that sensitivity (true positive rate) = 95%, specificity (true negative rate) = 95%, and prevalence = 5%.

2. Compute the confusion matrix

Assume a population of 10,000: 500 bad, 9,500 good. Calculate true positives (475), false negatives (25), true negatives (9,025), false positives (475).

3. Calculate PPV and NPV

PPV = TP / (TP + FP) = 475 / (475 + 475) = 50%. NPV = TN / (TN + FN) = 9025 / (9025 + 25) = 99.7%.

4. Analyze prevalence shift

Show that as prevalence increases, PPV increases and NPV decreases. For example, if prevalence = 50%, PPV = 95% and NPV = 95%.

5. Discuss base rate implications

Explain that when prevalence is low, even a highly specific test yields many false positives, so PPV is low. This is the base rate fallacy.

Key Points to Mention

Sensitivity and specificity are properties of the test, while PPV and NPV depend on prevalence.
Base rate fallacy: ignoring prevalence leads to overestimating the probability of a positive result being true.
The trade-off between false positives and false negatives depends on the cost of each error.
In product analytics, low prevalence scenarios (e.g., fraud detection) require high precision to avoid user friction.
Bayes' theorem can be used to compute PPV and NPV directly.
Communicate results in terms of business impact, not just numbers.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.