Citibank·Data Scientist·Technical Phone Screen·Senior

Senior

Jul 2026

Summary

Citibank data scientist interview that went deep into model risk territory. The whole thing revolved around one big technical question on PD model validation, and they clearly expected you to know the full lifecycle, not just throw out buzzwords.

Questions Asked (1)

Walk me through how you would validate a newly developed probability of default model, covering data partitioning, discrimination metrics, calibration, stability monitoring, backtesting, challenger models, and documentation.

Data ModelingTechnical Trade-offsProduct Analytics & Metrics

Author's notes

This is a lot to cover in one question and I kind of panicked trying to sequence it all.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Structure your answer around the model risk management lifecycle, emphasizing regulatory expectations (e.g., SR 11-7, Basel) and practical validation techniques. Start with data and methodology review, then cover quantitative performance (discrimination, calibration, stability), and finish with governance (backtesting, challengers, documentation).

Pro tip: Highlight that validation is not just a statistical exercise but a risk management process—mention that you would assess both conceptual soundness and ongoing monitoring, and that you would tailor tests to the model's use case (e.g., regulatory capital vs. pricing).

1. Data and Methodology Review

Verify data quality, partitioning (train/validation/test, out-of-time), and that the model development methodology aligns with best practices and regulatory guidance.

2. Discrimination and Calibration Assessment

Evaluate the model's ability to rank-order risk (e.g., AUC, Gini, KS) and the accuracy of predicted probabilities (e.g., Hosmer-Lemeshow, calibration plots, Brier score).

3. Stability and Backtesting

Test population stability (PSI) and characteristic stability, and backtest predictions against realized defaults using statistical tests (e.g., binomial, Jeffreys).

4. Challenger Models and Benchmarking

Develop or review challenger models (e.g., alternative algorithms, simpler models) to benchmark performance and assess whether the incumbent model remains fit for purpose.

5. Documentation and Governance

Ensure comprehensive documentation of validation findings, limitations, and recommendations, and present to model risk committee for approval and ongoing monitoring plan.

Key Points to Mention

Data partitioning: use out-of-time and out-of-sample validation to avoid overfitting and assess temporal robustness.
Discrimination metrics: AUC, Gini, KS, and lift charts; compare against benchmarks and challenger models.
Calibration: Hosmer-Lemeshow test, calibration plots, Brier score, and predicted vs. observed default rates.
Stability monitoring: Population Stability Index (PSI), characteristic stability, and monitoring triggers.
Backtesting: statistical tests (binomial, Jeffreys, traffic light approach) and comparison of predicted vs. actual defaults.
Challenger models: develop alternative models (e.g., logistic regression vs. machine learning) to benchmark and ensure the incumbent remains best-in-class.
Documentation: validation report, findings, limitations, and recommendations; align with SR 11-7 and internal model risk policies.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.