Snapchat·Machine Learning Engineer·Onsite - System Design / Architecture·Senior

Senior

Apr 2026

Summary

Snapchat ML engineer interview focused entirely on system design for a video recommendation pipeline. Pretty deep on the inter-service communication side, not just the usual 'describe your ML model' stuff.

Questions Asked (1)

Design a video recommendation system end-to-end, with specific attention to what data is passed between services. Identify the core services (candidate generation, feature service, ranker, re-ranker, serving/response builder, logging) and for each pair, define the payload, format/schema, latency budget, consistency guarantees, ordering guarantees, caching strategy, and how you'd minimize cross-service bandwidth.

System DesignTechnical Trade-offsData Modeling

Author's notes

This one went longer than I expected.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by clarifying requirements and scale, then walk through the end-to-end pipeline service by service, explicitly defining the data contract between each pair. For each hop, cover payload schema, latency budget, consistency/ordering guarantees, caching, and bandwidth optimization, and tie choices back to Snapchat's low-latency, mobile-first constraints.

Pro tip: Anchor every design decision to a concrete latency budget (e.g., 200ms total p99) and show how you'd measure and enforce it with per-hop deadlines and fallbacks; this demonstrates production maturity beyond just naming services.

1. Clarify requirements and constraints

Ask about scale (DAU, QPS), latency SLO, freshness needs, and device constraints. Establish a global latency budget and consistency requirements before designing.

2. Define core services and responsibilities

List candidate generation, feature service, ranker, re-ranker, serving/response builder, and logging. Briefly state each service's role and ownership of data.

3. Specify inter-service contracts

For each pair (e.g., candidate gen → feature service), define payload schema, format (Protobuf/Arrow), latency budget, consistency/ordering guarantees, caching strategy, and bandwidth minimization techniques.

4. Address cross-cutting concerns

Discuss failure handling, fallbacks, monitoring, and how logging feeds back into training. Explain trade-offs between consistency, latency, and cost.

5. Summarize and validate

Recap the end-to-end flow, confirm the design meets the SLO, and invite follow-up questions to show collaborative problem-solving.

Key Points to Mention

Use compact serialization (Protobuf, FlatBuffers, Arrow) and columnar formats for feature vectors to minimize bandwidth.
Set per-hop latency budgets (e.g., candidate gen 50ms, feature fetch 30ms, ranking 80ms) with deadlines and circuit breakers.
Cache aggressively at multiple levels: candidate sets, user features, and model outputs, with TTLs tuned to freshness needs.
Define consistency guarantees: eventual consistency for logging and feature updates, strong consistency for user state where needed.
Ordering guarantees: ensure ranker receives candidates in a stable order; use sequence numbers or timestamps for logging.
Minimize cross-service bandwidth by sending IDs instead of full payloads, batching requests, and using delta updates.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.