This was essentially seven questions rolled into one.
Structure your answer as a coherent narrative that starts with data splitting and imbalance handling, then moves through metric selection, threshold tuning, algorithm choices, and validation, ending with pseudocode. Emphasize that with such a low positive rate, careful validation and metric choice are critical to avoid misleading results.
Pro tip: Mention that you would use stratified splitting and consider grouping by time or entity to prevent leakage, and that you'd evaluate using precision-recall AUC and cost-sensitive metrics rather than accuracy or ROC AUC.
Split data into train/validation/test using stratification to preserve the 0.2% positive rate, and consider time-based splits if temporal patterns exist. Ensure no leakage by grouping related rows (e.g., same user) into the same split.
Address imbalance via techniques like class weighting, oversampling (e.g., SMOTE), or undersampling, but be cautious with synthetic data. Evaluate the impact on validation metrics and consider using anomaly detection if positives are extremely rare.
Choose metrics robust to imbalance: precision-recall AUC, F1, or cost-based metrics. Tune the decision threshold on validation data to optimize the business objective (e.g., minimize expected cost).
Start with a simple baseline (logistic regression) and move to tree-based ensembles (XGBoost, LightGBM) that handle imbalance well. Use early stopping and cross-validation. Provide pseudocode for the training loop with class weights and threshold tuning.
Use stratified k-fold cross-validation on training data, and a held-out test set for final evaluation. Monitor for overfitting and ensure the model generalizes to the minority class.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.