I answered it fine on the surface but stumbled when they pushed on why you'd need a separate validation set at all versus just using the test set for both.
Start by clarifying the project context (dataset size, class balance, temporal or group structure) and then propose a split strategy that avoids data leakage and ensures representative subsets. Discuss common ratios like 70/15/15 or 80/10/10, and mention advanced techniques such as stratified sampling, time-based splits, or group splits when applicable. Finally, explain how you would validate the split and monitor for distribution shifts.
Pro tip: Emphasize that the test set should be locked away until final evaluation and that any preprocessing (e.g., scaling, imputation) must be fit only on the training set to prevent leakage. Also, mention that for large datasets, a smaller test set (e.g., 1-2%) may suffice if it's statistically representative.
Ask about dataset size, feature types, class balance, and whether data has temporal or group dependencies. This determines the appropriate splitting method.
Select a method such as random split, stratified split, time-based split, or group split based on the data characteristics. For imbalanced data, use stratified sampling to preserve class ratios.
Decide on proportions for training, validation, and test sets. Common ratios are 70/15/15 or 80/10/10, but adjust based on dataset size and model complexity.
Perform the split before any preprocessing. Ensure that any transformations (e.g., normalization, feature engineering) are fit only on the training set and applied to validation and test sets.
Check that the distributions of key features and target are similar across splits. For production, consider setting up a holdout test set that is only used once, and monitor for data drift over time.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.