Upstart·Machine Learning Engineer·Onsite - Coding / Algorithms·Senior

Senior

Apr 2026

Summary

Upstart ML engineer interview with a hands-on case study using the California housing dataset. The whole thing was less about throwing features at a model and more about showing you understand distribution shift and can actually diagnose what's going wrong.

Questions Asked (1)

You're given the California housing dataset where train and test sets come from different income brackets, causing feature and label distribution shift. Build a regression model from scratch to predict house prices, then analyze and optimize it with that shift in mind.

Technical Trade-offsRoot Cause AnalysisData Modeling

Author's notes

I spent way too long on feature engineering early on and kind of assumed the model analysis part would be quick.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by acknowledging the distribution shift and its implications, then outline a plan to build a robust regression model. Focus on techniques to detect and mitigate shift, such as importance weighting or domain adaptation, and emphasize evaluation using metrics that account for shift.

Pro tip: Demonstrate awareness that in real-world ML, distribution shift is the norm, not the exception. Propose a monitoring strategy to detect shift in production and a retraining pipeline to adapt over time.

1. Understand and Quantify the Shift

Analyze the differences between train and test distributions using statistical tests (e.g., KS test) and visualize feature distributions. Quantify the shift's impact on a baseline model's performance.

2. Build a Baseline Model

Implement a simple regression model (e.g., linear regression) from scratch, train on the training set, and evaluate on both train and test sets to establish a performance baseline under shift.

3. Apply Shift Mitigation Techniques

Use importance weighting to reweight training samples based on their likelihood of appearing in the test distribution. Alternatively, consider domain adaptation methods or robust loss functions.

4. Optimize and Validate

Tune hyperparameters using cross-validation that respects the shift (e.g., group by income bracket). Evaluate using metrics robust to shift, such as weighted RMSE or performance on shifted subsets.

5. Monitor and Iterate

Propose a production monitoring plan to detect future shift and a strategy for periodic retraining or online learning to maintain performance.

Key Points to Mention

Distribution shift types: covariate shift (feature shift) and label shift (target shift), and how they apply here.
Importance weighting: estimating density ratio between test and train distributions to reweight samples.
Domain adaptation techniques: e.g., CORAL, adversarial training, or fine-tuning on target domain.
Evaluation metrics: use of weighted metrics, or evaluating on a held-out set that mimics test distribution.
Robustness: regularization, ensemble methods, or Bayesian approaches to handle uncertainty.
Production considerations: monitoring for shift, retraining triggers, and feedback loops.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.