TikTok·Data Scientist·Technical Phone Screen·Senior

Senior

May 2026

Summary

TikTok data scientist interview with a single monster question covering basically every dimension of experiment design you can think of. The kind of question where you finish answering and still aren't sure if you got half of it right.

Questions Asked (1)

You're launching a redesigned onboarding flow expected to lift Day-7 activation and potentially trigger network effects and weekly seasonality. Design a full experiment plan covering: hypothesis and metric definitions with attribution windows, randomization unit and interference handling, sample size and power analysis accounting for seasonality, variance reduction techniques, SRM detection, sequential monitoring and alpha spending, ramp plan with novelty effect controls, diagnostics for noncompliance and bot traffic, spillover detection and bias quantification, and a decision framework when primary and guardrail metrics conflict.

A/B Testing & ExperimentationProduct Analytics & Metrics

Author's notes

This one took me a second to process because it's not really one question, it's like ten questions stapled together.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Structure your answer around the experiment lifecycle: design, execution, monitoring, and decision-making. Emphasize how you would handle TikTok's unique challenges like network effects and seasonality, and show a clear decision framework for metric conflicts.

Pro tip: Demonstrate maturity by acknowledging that perfect experiments are rare; focus on quantifying and mitigating biases rather than eliminating them. Mention that you would pre-register the analysis plan and use a holdout group to measure long-term effects.

1. Define Hypothesis and Metrics

Clearly state the hypothesis (e.g., redesigned onboarding increases Day-7 activation by X%). Define primary metric (Day-7 activation) and guardrail metrics (e.g., retention, engagement, revenue). Specify attribution windows (e.g., 7-day) and how you'll handle network effects (e.g., cluster randomization).

2. Design Experiment and Power Analysis

Choose randomization unit (user-level or cluster-level to handle interference). Calculate sample size using power analysis, accounting for seasonality (e.g., weekly patterns) and expected effect size. Consider variance reduction techniques like CUPED or stratification.

3. Monitor and Detect Issues

Implement sequential monitoring with alpha spending (e.g., O'Brien-Fleming boundaries) to allow early stopping. Set up SRM checks, bot traffic filters, and noncompliance diagnostics. Detect spillover via network analysis or geo experiments and quantify bias.

4. Ramp and Control Novelty

Plan a gradual ramp (e.g., 1%, 5%, 10%, 50%) with holdout groups to measure novelty effects. Monitor metrics over time to distinguish novelty from true effect. Use a long-term holdout to assess sustained impact.

5. Decision Framework for Metric Conflicts

If primary metric improves but guardrails degrade, weigh trade-offs using pre-defined thresholds (e.g., guardrail must not drop by more than Y%). Consider business impact, statistical significance, and qualitative insights. Decide to launch, iterate, or abandon.

Key Points to Mention

Attribution windows: align with user behavior cycle (e.g., 7-day for activation) and consider multiple windows for robustness.
Interference handling: use cluster randomization (e.g., by geography or social graph clusters) to mitigate network effects.
Seasonality: incorporate weekly seasonality in power analysis by using stratified randomization or time-based blocking.
Variance reduction: apply CUPED using pre-experiment data to increase sensitivity.
SRM detection: monitor sample ratio with chi-squared test; if SRM occurs, investigate logging or randomization bugs.
Alpha spending: use sequential testing with alpha spending functions to control Type I error while allowing interim looks.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.