TikTok·Data Scientist·Technical Phone Screen·Senior

Senior

May 2026

Summary

TikTok data scientist interview with a product experimentation question. Pretty focused session, just one meaty scenario about A/B testing a homepage feature. The kind of question that feels manageable until you realize how many layers they actually want you to cover.

Questions Asked (1)

The product team runs an A/B test adding a new recommendation carousel to the homepage. What primary and secondary metrics would you track, and how would you decide whether to ship it to everyone?

A/B Testing & ExperimentationProduct Analytics & Metrics

Author's notes

I started with click-through rate as the primary metric which felt right, then conversion as secondary.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by defining the primary metric as the one that directly measures the carousel's impact on the core product goal, such as user engagement or watch time. Then outline secondary metrics that capture potential trade-offs, guardrails, and long-term effects. Finally, explain a decision framework that balances statistical significance, practical significance, and business impact.

Pro tip: Emphasize that at TikTok, the primary metric should align with the 'For You' experience—like total watch time or session depth—and always include guardrail metrics to catch negative side effects, such as reduced content diversity or user fatigue.

1. Define the primary metric

Choose a metric that directly reflects the carousel's intended impact on the core user experience, such as average watch time per user or daily active users.

2. Identify secondary and guardrail metrics

Select metrics that capture trade-offs (e.g., click-through rate on carousel items) and guardrails (e.g., bounce rate, content diversity, user reports) to ensure no harm.

3. Set up the experiment and analyze results

Ensure proper randomization, sample size, and duration. Analyze both statistical significance (p-value, confidence intervals) and practical significance (effect size, business impact).

4. Consider long-term and heterogeneous effects

Look beyond the experiment window for novelty effects and segment by user cohorts (e.g., new vs. existing users) to see if the carousel benefits some groups more.

5. Make a ship decision

Ship if the primary metric improves significantly without guardrail regressions, and if the lift justifies engineering and maintenance costs. Otherwise, iterate or abandon.

Key Points to Mention

Primary metric should be tied to the product's north star, e.g., watch time or session length.
Secondary metrics include engagement metrics (CTR, likes, shares) and guardrails (bounce rate, unfollows, reports).
Use statistical tests (t-test, bootstrapping) and check for novelty effects.
Consider network effects and content diversity, especially for a recommendation system.
Evaluate practical significance: is the lift large enough to matter for the business?
Segment by user demographics or behavior to detect heterogeneous treatment effects.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.