← TikTok Interview Insights

TikTok·Machine Learning Engineer·Onsite - System Design / Architecture·Senior

SeniorPrefer not to say
May 2026

Summary

TikTok ML engineer interview that was basically one big system design deep-dive on LLM-powered recommendation systems. Covered a lot of ground fast, from retrieval architectures to evaluation, and I felt underprepared for how far into the weeds they wanted to go.

Questions Asked (5)

Q1

How would you incorporate large language models into a recommendation system, and what use cases would you prioritize?

System DesignProduct Sense & IdeationTechnical Trade-offs
Author's notes

I started with metadata enrichment and cold-start handling because those felt like the safest bets to explain quickly.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by clarifying the recommendation system's current architecture and business goals, then propose LLM integration points that address specific pain points (e.g., cold-start, semantic understanding, explainability). Prioritize use cases by impact and feasibility, and discuss trade-offs like latency, cost, and scalability.

Pro tip: Emphasize hybrid approaches where LLMs augment rather than replace existing systems, and highlight how you'd measure success with online metrics (e.g., CTR, watch time) to ensure business impact.

1. Understand the Current System and Goals

Ask clarifying questions about the existing recommendation pipeline, data available, and key business metrics (e.g., user engagement, retention). Identify pain points where LLMs could add value.

2. Identify LLM Integration Points

Map LLM capabilities (e.g., semantic understanding, text generation, reasoning) to stages of the recommendation funnel: candidate generation, ranking, and post-ranking. Consider both content-based and collaborative filtering enhancements.

3. Prioritize Use Cases

Evaluate use cases by potential impact (e.g., improving cold-start recommendations, generating explanations) and feasibility (e.g., latency constraints, cost). Prioritize quick wins and high-impact areas like semantic embeddings for retrieval.

4. Address Technical Trade-offs

Discuss trade-offs: latency vs. quality (e.g., using distilled models), cost vs. performance (e.g., caching, batching), and offline vs. online evaluation. Propose a hybrid architecture where LLMs handle specific tasks.

5. Define Evaluation and Iteration Plan

Outline offline metrics (e.g., recall@k, NDCG) and online A/B tests (e.g., CTR, watch time). Suggest a phased rollout to monitor performance and iterate.

Key Points to Mention

  • Use LLMs for semantic embeddings to improve content understanding and cold-start recommendations.
  • Generate natural language explanations for recommendations to increase user trust and engagement.
  • Employ LLMs for query understanding and intent detection in search-based recommendations.
  • Leverage LLMs for data augmentation and synthetic data generation to enhance training.
  • Address latency and cost by using smaller distilled models or caching frequent queries.
  • Evaluate with both offline metrics and online A/B tests focusing on business KPIs.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.

Q2

Walk me through different architectural patterns for integrating LLMs into a recommender, such as using the LLM as a feature generator, a reranker, or an orchestration agent.

System DesignTechnical Trade-offs
Author's notes

This is where I felt most comfortable.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Structure your answer by first outlining the main architectural patterns (feature generator, reranker, orchestration agent) and then comparing them on dimensions like latency, cost, and impact. Emphasize how each pattern fits into a typical recommender pipeline and discuss trade-offs in the context of a large-scale platform like TikTok.

Pro tip: Quantify trade-offs with concrete examples (e.g., 'LLM as feature generator adds ~100ms latency but can improve cold-start recommendations by 15%') to show you understand real-world constraints. Also, mention hybrid approaches where multiple patterns are combined for different stages of the funnel.

1. Clarify the recommender pipeline

Briefly describe the standard stages: candidate generation, ranking, and re-ranking. This sets the context for where LLMs can be integrated.

2. Pattern 1: LLM as feature generator

Explain how LLMs can generate embeddings or textual features from user/item data to enrich the input to traditional models. Discuss benefits like handling cold-start and drawbacks like latency and cost.

3. Pattern 2: LLM as reranker

Describe using LLMs to reorder a small set of top candidates from an initial ranker. Highlight advantages like improved relevance and personalization, and challenges like inference cost and latency.

4. Pattern 3: LLM as orchestration agent

Discuss using LLMs to dynamically select and coordinate multiple recommendation strategies or tools. Mention benefits like adaptability and complexity in control and evaluation.

5. Compare and recommend

Summarize trade-offs across patterns (latency, cost, impact, complexity) and suggest when to use each, possibly proposing a hybrid approach for different stages.

Key Points to Mention

  • Latency and cost constraints in large-scale systems like TikTok
  • Impact on cold-start and long-tail recommendations
  • Integration complexity with existing ML infrastructure
  • Evaluation metrics (e.g., CTR, watch time) and A/B testing considerations
  • Hybrid approaches combining multiple patterns
  • Scalability and real-time inference challenges

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.

Q3

How do you handle latency and cost constraints when LLMs are part of the serving path for recommendations?

System DesignTechnical Trade-offs
Author's notes

Caching was the first thing I said, which was fine, but I didn't structure it well.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by acknowledging the inherent tension between latency, cost, and quality in LLM-based recommendation serving. Then, outline a layered architecture that uses LLMs selectively (e.g., for high-value users or re-ranking) and employs caching, distillation, and quantization to meet constraints. Finally, discuss trade-offs and metrics to monitor, showing a balanced, production-oriented mindset.

Pro tip: Emphasize that you would first establish a baseline without LLMs and then incrementally add LLM components only where they provide clear ROI, measuring impact on both latency and cost per request. This demonstrates pragmatism and avoids over-engineering.

1. Clarify requirements and constraints

Ask about latency SLAs (e.g., p99 < 100ms), cost budgets (e.g., $ per 1k requests), and scale (QPS). Also understand the recommendation stage (retrieval, ranking, re-ranking) and user experience impact.

2. Design a hybrid architecture

Propose using LLMs only where they add unique value, such as re-ranking top-N candidates or generating explanations. Keep traditional models for retrieval and initial ranking to meet latency and cost.

3. Apply optimization techniques

Discuss model-level optimizations: distillation into smaller models, quantization (e.g., 8-bit), pruning, and caching frequent queries. Also consider batching and asynchronous processing for non-real-time parts.

4. Implement fallbacks and degradation

Design graceful degradation: if LLM latency spikes or cost exceeds budget, fall back to non-LLM models or cached results. Use circuit breakers and monitor key metrics.

5. Measure and iterate

Define metrics: p99 latency, cost per request, CTR, engagement. A/B test LLM components to ensure they justify their cost. Continuously optimize based on data.

Key Points to Mention

  • Latency budgets: p99 latency targets and how LLM inference adds 100ms+ per call, requiring careful placement.
  • Cost per inference: token-based pricing, GPU costs, and how caching or distillation reduces cost.
  • Model distillation: training a smaller model to mimic LLM outputs for faster, cheaper inference.
  • Caching strategies: caching LLM outputs for frequent queries or user segments to avoid redundant calls.
  • Hybrid serving: using LLMs only for re-ranking or high-value users, not for every request.
  • Fallback mechanisms: defaulting to traditional models if LLM latency or cost exceeds thresholds.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.

Q4

How would you design an evaluation framework for an LLM-augmented recommendation system, covering both offline metrics and online experiments?

A/B Testing & ExperimentationProduct Analytics & Metrics
Author's notes

Blanked a bit on the guardrails piece.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by framing the evaluation framework around the dual goals of offline validation and online experimentation, emphasizing the unique challenges of LLM-augmented systems such as generative outputs and delayed feedback. Then, outline a layered approach: offline metrics for rapid iteration, online A/B tests for causal impact, and guardrail metrics to catch regressions. Finally, highlight the importance of aligning metrics with business objectives and user experience.

Pro tip: Emphasize the need for counterfactual evaluation and human-in-the-loop validation to address the limitations of offline metrics for generative recommendations. Also, mention the importance of long-term holdout groups to measure delayed effects and avoid short-term metric myopia.

1. Define Objectives and Metrics

Clarify the goals of the LLM-augmented recommender (e.g., engagement, satisfaction, diversity) and map them to measurable offline and online metrics. Include both business metrics (CTR, watch time) and LLM-specific metrics (coherence, relevance).

2. Offline Evaluation

Use historical data to compute ranking metrics (NDCG, Recall@K) and generative metrics (BLEU, ROUGE, perplexity) where applicable. Incorporate human evaluation for subjective quality and counterfactual estimators to simulate online impact.

3. Online Experimentation

Design A/B tests with proper randomization, sample size, and duration. Define primary and guardrail metrics, and use interleaving or switchback tests for faster iteration. Monitor for novelty effects and segment by user cohorts.

4. Iterate and Validate

Continuously refine the framework by comparing offline predictions with online outcomes, and use techniques like offline replay or simulation to improve offline-online correlation. Establish a feedback loop for model updates.

5. Monitor and Guardrail

Implement real-time monitoring for system health, bias, and safety. Set up guardrail metrics (e.g., toxicity, diversity) to halt experiments if thresholds are breached, ensuring responsible AI deployment.

Key Points to Mention

  • Offline metrics: NDCG, Recall@K, MRR for ranking; BLEU, ROUGE, perplexity for generation; human evaluation for relevance and coherence.
  • Online metrics: CTR, watch time, conversion rate, user satisfaction (surveys), diversity, and novelty.
  • A/B testing best practices: randomization unit, sample size calculation, A/A tests, guardrail metrics, and sequential testing.
  • Counterfactual evaluation and offline replay to estimate online performance without live traffic.
  • Long-term holdout groups to measure delayed effects and avoid short-term optimization.
  • LLM-specific challenges: hallucination, bias, toxicity, and the need for human-in-the-loop evaluation.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.

Q5

What safety and content filtering considerations apply when LLMs are generating or influencing recommendation outputs?

Technical Trade-offsSystem Design
Author's notes

Shorter exchange but still tricky.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Frame your answer around a layered defense-in-depth strategy, covering input filtering, model behavior constraints, output moderation, and continuous monitoring. Emphasize the unique challenges of LLMs in recommendation systems, such as hallucination, bias amplification, and adversarial prompts, and how to balance safety with engagement and personalization.

Pro tip: Show you understand TikTok's specific context by mentioning how LLM-generated recommendations must comply with platform policies and local regulations, and that safety filters should be evaluated not just for accuracy but also for latency and scalability in a real-time system.

1. Identify LLM-specific risks in recommendations

Discuss risks like generating harmful or misleading content, amplifying biases, leaking private data, and being manipulated via prompt injection. Explain how these can affect recommendation outputs directly or indirectly.

2. Design input and prompt safeguards

Cover techniques such as sanitizing user inputs, using system prompts to constrain LLM behavior, and employing retrieval-augmented generation with vetted knowledge sources to ground outputs.

3. Implement output filtering and moderation

Describe multi-stage filters: rule-based checks for prohibited terms, ML classifiers for toxicity and NSFW content, and human-in-the-loop review for edge cases. Highlight the need for low-latency solutions.

4. Monitor and iterate with feedback loops

Explain how to set up real-time monitoring for safety metrics, A/B test filter thresholds, and incorporate user reports to continuously improve the system without stifling engagement.

5. Balance safety with personalization and engagement

Discuss trade-offs between aggressive filtering and recommendation quality, and propose methods like multi-objective optimization to maintain user experience while adhering to safety policies.

Key Points to Mention

  • Prompt injection and adversarial attacks on LLMs
  • Bias detection and mitigation in generated content
  • Real-time content moderation at scale (latency, throughput)
  • Regulatory compliance (e.g., COPPA, GDPR, DSA) and platform policies
  • Human-in-the-loop and appeals processes
  • Evaluation metrics for safety filters (precision, recall, false positive rate)

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.