TikTok·Data Scientist·Technical Phone Screen·Senior

SeniorPrefer not to say

Jun 2026Remote

Summary

TikTok data scientist interview focused heavily on experimentation design and causal inference. Two meaty questions back to back with no warmup, which felt a bit abrupt. The topics were squarely in the DS wheelhouse but the depth expected was real.

Questions Asked (2)

An A/B test changed a call-to-action button color from green to red and retention dropped. Walk through the diagnostics you'd run to figure out whether the result was driven by uneven traffic allocation or other experiment quality problems.

A/B Testing & ExperimentationRoot Cause Analysis

Author's notes

I started with sample-ratio mismatch, which felt like the obvious first move, and they nodded along.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by verifying the experiment's health: check for sample ratio mismatch (SRM) and ensure the randomization unit and analysis unit align. Then, if allocation is balanced, investigate other quality issues like novelty effects, instrumentation errors, or external factors before concluding the color caused the retention drop.

Pro tip: Always check for SRM first—it's the quickest way to catch allocation bugs. If SRM is present, the experiment is invalid, and you should avoid interpreting the metric change until fixed.

1. Check for Sample Ratio Mismatch (SRM)

Compare the observed traffic split to the expected split using a chi-square test. If the p-value is significant (e.g., <0.001), there's likely an allocation bug, and the experiment results are unreliable.

2. Validate Randomization and Assignment

Ensure the randomization unit (e.g., user ID) is correctly implemented and that users are consistently assigned to the same variant. Check for any leakage or cross-contamination between groups.

3. Inspect Data Quality and Instrumentation

Look for missing data, logging errors, or metric definition changes during the experiment. Verify that retention is measured consistently across both groups and that no tracking bugs skew the results.

4. Analyze Segment-Level Metrics and External Factors

Break down retention by key dimensions (e.g., device, geography, user tenure) to see if the drop is concentrated in a subgroup. Also, check for concurrent experiments or external events that could confound results.

5. Conclude and Decide Next Steps

If no quality issues are found, consider the result valid and investigate the causal mechanism (e.g., color psychology). If issues exist, fix them and rerun the experiment.

Key Points to Mention

Sample Ratio Mismatch (SRM) detection and its implications
Randomization unit consistency and potential assignment bugs
Data quality checks: missing data, logging errors, metric definitions
Segment analysis to identify heterogeneous treatment effects
External validity threats: novelty effects, concurrent experiments, seasonality
Statistical power and the risk of false positives/negatives

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.

You need to measure the causal effect of receiving negative reviews on a merchant's coupon repurchase rate. What data would you need and what methodology would you use to get an unbiased estimate?

A/B Testing & ExperimentationProduct Analytics & MetricsData Modeling

Author's notes

This one tripped me up more than I expected.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by clarifying the causal question and the ideal experiment (randomized negative reviews), then discuss why it's infeasible and propose an observational causal inference method like difference-in-differences or instrumental variables. Outline the required data (panel data on merchants, reviews, and repurchases) and key assumptions for unbiasedness.

Pro tip: Emphasize that the key challenge is confounding by merchant quality and customer satisfaction; suggest using a natural experiment or a valid instrument (e.g., random assignment of reviews to moderators) to isolate causal effect.

1. Clarify the causal question and ideal experiment

Define treatment (receiving negative reviews) and outcome (coupon repurchase rate). Discuss the gold standard: randomized experiment where negative reviews are randomly assigned, and why it's often infeasible.

2. Identify data requirements

List necessary data: merchant-level panel data on coupon repurchases, review timestamps and sentiment, merchant characteristics, customer demographics, and potential instruments (e.g., reviewer assignment).

3. Choose a causal inference method

Propose methods like difference-in-differences (if staggered review shocks), instrumental variables (if a valid instrument exists), or synthetic control. Discuss assumptions and how to test them.

4. Address confounding and robustness

Explain how to control for time-varying confounders (e.g., merchant promotions, seasonality) and conduct sensitivity analyses (e.g., placebo tests, negative controls).

5. Validate and interpret results

Check for heterogeneous effects (e.g., by merchant size), validate with falsification tests, and discuss limitations and practical implications for TikTok's coupon strategy.

Key Points to Mention

Randomized controlled trial (A/B test) as the ideal but often infeasible due to ethical/practical constraints
Difference-in-differences with staggered treatment timing and parallel trends assumption
Instrumental variables: using random assignment of reviews to moderators or exogenous review shocks
Propensity score matching or weighting to balance observed covariates
Panel data structure with merchant fixed effects to control for time-invariant unobserved heterogeneity
Sensitivity analysis and placebo tests to assess robustness of causal estimates

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.