Meta·Data Scientist·Technical Phone Screen
- You have a credit card fraud classifier deployed as a probability scorer. Given a base fraud rate of 0.20% across 1 million transactions, and asymmetric costs of $15 per false positive and $100 per false negative, compute the expected TP, FP, FN, TN counts and total cost for three candidate thresholds. Which threshold minimizes cost and what are the business trade-offs?
- How would you calibrate the model's probability scores, monitor for data or label drift over time, and periodically re-tune the operating threshold by segment such as country, merchant category, or transaction amount?
- What guardrail metrics would you set up to catch harmful side effects like a spike in declines for high-value customers, and how would you design an experiment to validate threshold changes without causing serious customer harm?
“This is the kind of question where the math looks straightforward but you can fumble it fast if you mix up base rates.”