Databricks·Data Scientist·Technical Phone Screen·Senior

Senior

Jul 2026

Summary

Databricks data scientist interview with a meaty technical question on evaluation metrics. Just one question but it had six sub-parts and they clearly wanted you to go deep, not just recite definitions.

Questions Asked (1)

You've trained a binary classifier that outputs predicted probabilities. Walk through ROC-AUC vs PR-AUC: what each curve actually plots, how class imbalance affects them, when they disagree, which you'd prefer for model selection, how you'd think about it if the business goal is catching as many positives as possible within a fixed review budget, and what practical pitfalls exist with either metric.

Product Analytics & MetricsTechnical Trade-offsA/B Testing & Experimentation

Author's notes

This is the kind of question where you think you know it and then halfway through you realize you're just reciting things you memorized.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by clearly defining what ROC-AUC and PR-AUC plot and how they are computed, then contrast their behavior under class imbalance. Use the business context of a fixed review budget to argue for PR-AUC or precision@k, and discuss practical pitfalls like threshold selection and metric gaming.

Pro tip: Emphasize that PR-AUC is more informative when the positive class is rare, but always tie the choice back to the business objective—e.g., if the cost of false positives is low and you have a fixed budget, precision-recall curves or lift charts are more actionable than ROC-AUC.

1. Define the curves

Explain that ROC-AUC plots True Positive Rate vs. False Positive Rate across thresholds, while PR-AUC plots Precision vs. Recall. Clarify that PR-AUC is often computed as average precision.

2. Impact of class imbalance

Discuss how ROC-AUC can be overly optimistic when negatives dominate because FPR is diluted, whereas PR-AUC directly reflects performance on the positive class and is more sensitive to imbalance.

3. When they disagree

Describe scenarios where ROC-AUC is high but PR-AUC is low (e.g., many false positives among a small positive set) and vice versa, and explain why this happens.

4. Model selection and business goal

Argue that for catching positives within a fixed review budget, you should optimize for precision at a fixed recall or recall at a fixed precision, using PR curves or precision@k. PR-AUC is generally preferred for imbalanced data, but the final choice depends on the cost of errors and operational constraints.

5. Practical pitfalls

Mention pitfalls: ROC-AUC can hide poor performance on the minority class; PR-AUC can be unstable with few positives; both are threshold-agnostic and may not reflect the chosen operating point; and they can be gamed by overfitting to the metric.

Key Points to Mention

ROC-AUC is threshold-independent and measures ranking ability across all thresholds, but its interpretation is less intuitive for imbalanced data.
PR-AUC focuses on the positive class and is more informative when positives are rare, but it can be noisy if the number of positives is small.
The baseline for PR-AUC is the prevalence of the positive class, not 0.5, which is a common misconception.
For a fixed review budget, you care about precision at a specific recall (or recall at a specific precision), so you should evaluate the PR curve at that operating point.
Lift curves and cumulative gains charts are also useful for budget-constrained problems.
Always consider the business cost of false positives vs. false negatives and the practical constraints of the review process.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.