Shopify·Machine Learning Engineer·Onsite - System Design / Architecture·Senior

SeniorPrefer not to say

Jul 2026Remote

Summary

Shopify ML engineer interview with a meaty open-ended system design question around building a package delivery ETA model. Single question but it sprawled into six or seven distinct sub-problems, which made it feel more like a 45-minute design session than a traditional interview.

Questions Asked (1)

Design an end-to-end ML system to predict package delivery ETA, covering target definition, model choice, feature engineering, handling missing scan events, offline and online evaluation, prediction intervals, and production monitoring.

System DesignTechnical Trade-offsProduct Analytics & Metrics

Author's notes

This one is deceptively wide.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by clarifying the business objective and defining the target as a distribution of arrival times rather than a point estimate, then walk through the ML lifecycle: data sources, feature engineering, model selection, handling missing scans, offline/online evaluation, prediction intervals, and monitoring. Emphasize trade-offs between accuracy, latency, and interpretability, and tie each decision back to user impact and operational constraints.

Pro tip: Frame the problem as probabilistic forecasting with prediction intervals, and highlight how you'd handle missing scan events by treating them as informative missingness—using models that natively handle missing data or imputing with uncertainty. Also, mention that you'd start with a simple baseline (e.g., historical median per route) to set a performance bar before moving to complex models.

1. Define the target and success metrics

Clarify whether ETA is for the entire delivery or per-leg, and define the target as a probability distribution over arrival times (e.g., quantiles). Align offline metrics (e.g., pinball loss) with online business metrics (e.g., customer satisfaction, reduced WISMO calls).

2. Data sources and feature engineering

Identify data sources: scan events, route info, historical delivery times, weather, traffic, carrier performance. Engineer features like time since last scan, distance to destination, historical median for similar routes, and temporal patterns (day of week, holidays).

3. Model selection and handling missing scans

Choose a model that outputs prediction intervals (e.g., quantile regression, gradient boosting with quantile loss, or Bayesian methods). Handle missing scan events by using models that accept missing values (e.g., XGBoost) or by imputing with uncertainty, and include missingness indicators as features.

4. Offline and online evaluation

Offline: use time-based splits, evaluate with pinball loss and calibration plots. Online: A/B test with metrics like ETA accuracy, coverage of prediction intervals, and business KPIs. Consider delayed feedback and counterfactual evaluation.

5. Production monitoring and iteration

Monitor data drift, prediction drift, and performance metrics (e.g., interval coverage, MAE). Set up alerts for degradation, and implement a feedback loop to retrain models periodically. Also monitor system latency and throughput.

Key Points to Mention

Target definition: probabilistic forecasting with prediction intervals (e.g., 10th, 50th, 90th percentiles) to capture uncertainty.
Feature engineering: temporal features, route characteristics, carrier historical performance, and real-time signals like weather and traffic.
Handling missing scan events: treat missingness as informative, use models that handle missing data natively, or impute with uncertainty and add missingness indicators.
Model choice: quantile regression, gradient boosting (e.g., LightGBM with quantile objective), or Bayesian approaches for uncertainty quantification.
Offline evaluation: pinball loss, calibration, and coverage of prediction intervals; use time-based validation to mimic production.
Online evaluation and monitoring: A/B testing with business metrics, monitoring for data/prediction drift, and setting up alerts for model degradation.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.