Capital One·Data Scientist·Technical Phone Screen·Senior

Senior

Jun 2026

Summary

Capital One data scientist interview, one big technical question that basically ate the whole session. It was a credit risk modeling deep dive and they wanted you to think through the entire pipeline from offline eval to production monitoring. Pretty intense for a single question.

Questions Asked (1)

You're deploying a probability-of-default model for a consumer lender with a 1% base rate and asymmetric misclassification costs ($1,200 for a missed defaulter, $60 for a wrongly declined good applicant). The regulator cares about stability and explainability. Walk through your three top evaluation priorities, then design the full pipeline: offline evaluation with metric choices, threshold selection under a decline-rate constraint, out-of-time backtesting, a live champion-challenger setup with guardrails, and a production monitoring plan including what you'd do if calibration drifts but rank ordering holds.

Product Analytics & MetricsTechnical Trade-offsSystem Design

Author's notes

This question is enormous and I think I underestimated how much they wanted actual numbers rather than a framework tour.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by framing the three evaluation priorities around the business and regulatory context: cost-sensitive performance, stability, and explainability. Then walk through the pipeline stages in order, explicitly linking each design choice to the 1% base rate, asymmetric costs, and decline-rate constraint. Finish with monitoring and a clear decision rule for calibration drift with stable rank ordering.

Pro tip: Emphasize that with a 1% base rate, precision is naturally low, so you should focus on lift and cost savings rather than accuracy. Also, mention that you would validate the cost assumptions with finance and legal, as they drive threshold selection.

1. Define evaluation priorities

Prioritize cost-sensitive performance (e.g., expected cost or profit), stability over time and across segments, and explainability for regulatory compliance. Explain how each priority influences metric and model choices.

2. Design offline evaluation

Choose metrics that align with business costs: expected cost, cost-sensitive AUC, lift at the operating threshold, and calibration metrics (Brier score, calibration curve). Use stratified sampling or weighting to handle the 1% base rate.

3. Select threshold under decline-rate constraint

Optimize the threshold to minimize expected cost subject to a maximum decline rate (e.g., 5%). Plot the cost and decline rate as functions of threshold to find the feasible optimum.

4. Backtest out-of-time and set up champion-challenger

Perform out-of-time backtesting on a holdout period to assess stability and performance. For live deployment, use a champion-challenger framework with guardrails: limit challenger exposure, monitor key metrics, and define rollback criteria.

5. Monitor production and handle calibration drift

Monitor input drift, score distribution, calibration, and rank ordering. If calibration drifts but rank ordering holds, recalibrate the model (e.g., Platt scaling or isotonic regression) using recent data, and consider a segmented recalibration if drift is segment-specific.

Key Points to Mention

Cost-sensitive metrics: expected cost = 1200*FN + 60*FP, and cost-sensitive AUC or lift.
Handling class imbalance: use stratified sampling, class weights, or focal loss; avoid accuracy.
Threshold selection: minimize expected cost subject to decline rate ≤ target; use validation set.
Out-of-time backtesting: train on past, test on future; check stability of performance and calibration.
Champion-challenger guardrails: limit exposure (e.g., 5-10%), monitor key metrics, define rollback triggers.
Calibration drift with stable rank ordering: recalibrate (Platt/isotonic) rather than retrain; monitor segment-level drift.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.