Capital One·Data Scientist·Technical Phone Screen·Senior

Senior

May 2026

Summary

Capital One data scientist interview with a deeply technical ML question focused on imbalanced classification, pipeline design, and threshold selection. Single question but it covered a lot of ground, felt more like a take-home problem squeezed into a conversation.

Questions Asked (1)

Build a scikit-learn Pipeline for a heavily imbalanced binary label (~1% positives). The pipeline should handle imputation, one-hot encoding for categoricals, and standard scaling for numerics, then train classifiers robust to class imbalance. Use GroupKFold with 5 folds grouped by user_id. Compare LogisticRegression with class_weight='balanced' against HistGradientBoostingClassifier, calibrate the better model with CalibratedClassifierCV on an inner fold, and report cross-validated ROC-AUC and PR-AUC. On a held-out fold, find the smallest probability threshold achieving precision >= 0.50 and report the corresponding recall, F1, and expected alert volume per 100k users. Explain how you prevent threshold selection from leaking into cross-validation.

Technical Trade-offsProduct Analytics & MetricsAlgorithms & Data Structures

Author's notes

The leakage question at the end is what actually tripped me up.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Structure your answer around the pipeline architecture, the nested cross-validation strategy for unbiased threshold selection, and the evaluation metrics tailored to imbalance. Emphasize how GroupKFold prevents user leakage and how the inner fold calibration and threshold selection avoid optimistic bias. Conclude with the trade-off between precision and alert volume, showing business awareness.

Pro tip: Mention that PR-AUC is more informative than ROC-AUC for imbalanced data, and that threshold selection must be done on a separate validation set or via nested CV to avoid leakage. Also, highlight that calibration is crucial for threshold-based decisions, especially with class_weight='balanced' which distorts probabilities.

1. Pipeline Construction

Build a scikit-learn Pipeline with ColumnTransformer: impute missing values (median for numeric, most frequent for categorical), one-hot encode categoricals, and standard scale numerics. Then append the classifier.

2. Model Training and Comparison

Use GroupKFold with 5 folds grouped by user_id to train and evaluate LogisticRegression (class_weight='balanced') and HistGradientBoostingClassifier. Compute cross-validated ROC-AUC and PR-AUC for each, and select the better model based on PR-AUC.

3. Calibration and Threshold Selection

For the better model, perform nested cross-validation: within each outer training fold, use an inner GroupKFold to calibrate with CalibratedClassifierCV and select the smallest probability threshold achieving precision >= 0.50. Apply this threshold to the outer test fold to compute recall, F1, and alert volume per 100k users.

4. Leakage Prevention

Ensure threshold selection is done only on inner folds, never on the outer test fold. This prevents optimistic bias and mimics real-world deployment where thresholds are set on historical data and applied to future data.

5. Reporting and Business Interpretation

Report average cross-validated ROC-AUC and PR-AUC, and the threshold metrics from outer folds. Discuss the trade-off between precision and alert volume, and how the chosen threshold aligns with business capacity.

Key Points to Mention

GroupKFold prevents data leakage by ensuring the same user_id does not appear in both training and validation folds.
PR-AUC is preferred over ROC-AUC for imbalanced datasets because it focuses on the minority class performance.
CalibratedClassifierCV with method='sigmoid' or 'isotonic' corrects probability estimates, which is essential for threshold-based decisions.
Nested cross-validation separates hyperparameter tuning and threshold selection from model evaluation to avoid optimistic bias.
class_weight='balanced' in LogisticRegression adjusts weights inversely proportional to class frequencies, but may require calibration for probability outputs.
Alert volume per 100k users is calculated as (number of predicted positives / total users) * 100,000, providing a business-relevant metric.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.