Shopify·Machine Learning Engineer·Onsite - System Design / Architecture·Senior

Senior

Jul 2026

Summary

Shopify ML engineer interview, one meaty system design question about building a product ranking model end to end. Pretty open-ended, which I liked but also found stressful because there's no obvious stopping point.

Questions Asked (1)

You're building a product ranking model for an e-commerce marketplace. Given a user, some context, and a set of candidate products, how would you rank them so the most relevant items show up first? Walk through data collection, label definition, features, model choices, training, and offline/online evaluation.

System DesignA/B Testing & ExperimentationTechnical Trade-offs

Author's notes

This is a lot of ground to cover and I think I spread myself too thin.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Structure your answer as an end-to-end ML system design, starting from business objectives and data collection, then moving through label definition, feature engineering, model selection, training, and evaluation. Emphasize the iterative nature of ranking systems and the importance of aligning offline metrics with online business metrics. Highlight trade-offs and practical considerations specific to e-commerce marketplaces like Shopify.

Pro tip: Always tie your technical choices back to business impact—e.g., how ranking improvements increase conversion or GMV—and mention the importance of guardrail metrics to avoid optimizing relevance at the expense of diversity or seller fairness.

1. Clarify Objectives and Data Collection

Define the business goal (e.g., maximize conversion or revenue) and identify data sources: user interactions (clicks, purchases), product metadata, user profiles, and context (time, device). Discuss logging and feedback loops.

2. Define Labels and Features

Choose labels like click, add-to-cart, or purchase, and handle biases (position bias, selection bias). Engineer features: user features (history, demographics), product features (category, price, popularity), context features (time, location), and interaction features.

3. Select and Train Model

Choose a model architecture (e.g., learning-to-rank, two-tower, or gradient boosted trees) based on scale and latency. Train with appropriate loss (e.g., pairwise or listwise) and validate using time-based splits to mimic production.

4. Offline Evaluation

Evaluate with ranking metrics (NDCG, MAP, MRR) and business metrics (precision@k). Use counterfactual or unbiased evaluation techniques to mitigate biases. Compare against baselines.

5. Online Evaluation and Iteration

Deploy via A/B test, monitor online metrics (CTR, conversion, revenue) and guardrails (latency, diversity). Iterate based on results, and consider multi-armed bandits for exploration.

Key Points to Mention

Handling position bias and selection bias in labels (e.g., using inverse propensity scoring or click models).
Feature engineering for e-commerce: user history, product embeddings, contextual signals, and real-time features.
Model choices: learning-to-rank (LambdaMART, DNN), two-tower retrieval, and trade-offs between latency and accuracy.
Offline evaluation metrics: NDCG, MAP, and how to align them with online business metrics.
Online evaluation: A/B testing, guardrail metrics, and avoiding feedback loops.
Scalability and latency considerations for serving in a marketplace with millions of products.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.