Upstart·Data Scientist·Onsite - System Design / Architecture·Senior

SeniorPrefer not to say

May 2026

Summary

Upstart data scientist interview that went pretty deep into ML system design for ads. The question was broad enough that you could spend an hour on any single piece of it, which made scoping feel like half the challenge.

Questions Asked (1)

Design a complete machine learning system for ad selection on a platform serving millions of users in real time. Walk through data collection, labeling, feature engineering, model choice, exploration vs. exploitation, online and offline evaluation, latency constraints, and scalability.

System DesignTechnical Trade-offsA/B Testing & Experimentation

Author's notes

This is a monster of a question.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by clarifying the business objective (e.g., maximize clicks, conversions, or revenue) and constraints (latency, scale). Then walk through the ML pipeline end-to-end, emphasizing trade-offs at each stage and how you would measure success both offline and online.

Pro tip: Always tie technical decisions back to business metrics and explicitly discuss how you would handle cold-start and delayed feedback, which are critical in real-time ad systems.

1. Clarify Requirements and Constraints

Ask about the objective (CTR, CVR, revenue), latency budget (e.g., <100ms), scale (QPS, number of ads), and data availability. This ensures your design is aligned with business needs.

2. Data Collection and Labeling

Describe logging user interactions (impressions, clicks, conversions) and how to handle delayed feedback. Discuss labeling strategies (e.g., using clicks as positive labels) and dealing with class imbalance.

3. Feature Engineering and Model Selection

Outline features (user, ad, context, cross features) and model choices (e.g., logistic regression for baseline, gradient boosted trees, or deep learning for scalability). Mention embedding techniques for high-cardinality features.

4. Exploration vs. Exploitation and Online Evaluation

Explain how to balance exploration (e.g., epsilon-greedy, Thompson sampling) with exploitation. Describe A/B testing, interleaving, and how to measure incremental lift.

5. Latency, Scalability, and Offline Evaluation

Discuss serving architecture (e.g., model caching, feature store), latency optimizations (e.g., approximate nearest neighbors), and offline metrics (AUC, log loss) with proper validation (time-based splits).

Key Points to Mention

Handling delayed feedback and using techniques like importance weighting or survival analysis.
Feature store for consistent online/offline features and low-latency serving.
Exploration strategies to avoid feedback loops and ensure unbiased evaluation.
Offline evaluation pitfalls: using time-based splits and avoiding data leakage.
Scalability considerations: distributed training, model compression, and real-time inference optimizations.
Business metrics vs. ML metrics: aligning offline improvements with online KPIs.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.