TikTok·Data Scientist·Technical Phone Screen·Senior

Senior

Jun 2026

Summary

TikTok data scientist interview that was basically one long scenario question about multi-arm testing across acquisition channels. Dense, technical, and they clearly wanted to see if you could hold the whole thing together from metric definition all the way through post-hoc analysis.

Questions Asked (1)

You have four acquisition channels (YouTube, Google Search, Facebook, direct mail) and a fixed marketing budget. How would you design a multi-arm experiment to compare cost-per-conversion efficiency across all four? Walk through your metric definition, hypotheses, statistical test choice, sample size and budget split, test duration, and any follow-up analyses you'd run after the main result.

A/B Testing & ExperimentationProduct Analytics & MetricsTechnical Trade-offs

Author's notes

This is a lot to hold in your head at once and I kind of fumbled the ordering.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Frame the problem as a multi-arm bandit or fixed-horizon A/B/n test with cost-per-conversion as the primary metric, then systematically walk through metric definition, hypotheses, power analysis, budget allocation, and follow-up analyses. Emphasize practical constraints like budget limits, unequal variances, and the need for sequential monitoring.

Pro tip: Mention that cost-per-conversion is a ratio metric with high variance, so you'd use a log transformation or a Poisson/negative binomial model, and consider a sequential test to allow early stopping for cost efficiency.

1. Define the metric and hypotheses

Clearly define cost-per-conversion (total spend / number of conversions) and state null and alternative hypotheses for comparing all four channels. Consider whether you're testing for any difference or specific pairwise differences.

2. Choose the statistical test and power analysis

Select an appropriate test (e.g., ANOVA on log-transformed costs, Kruskal-Wallis, or a generalized linear model) and conduct a power analysis to determine required sample size per arm, accounting for expected variance and minimum detectable effect.

3. Allocate budget and design the experiment

Decide on a budget split across channels (equal or weighted by expected performance) and set a test duration that balances statistical power with practical constraints like seasonality and budget depletion.

4. Run the test and monitor

Execute the experiment, monitor for data quality and early stopping criteria (if using sequential testing), and ensure no budget overspend or under-spend in any arm.

5. Analyze results and follow-up

Perform the main test, then conduct pairwise comparisons with multiple testing correction, segment analyses (e.g., by audience or time), and sensitivity checks. Recommend next steps like reallocating budget or running a confirmatory test.

Key Points to Mention

Cost-per-conversion is a ratio metric; use delta method or bootstrap for confidence intervals.
Consider unequal variances and non-normal distributions; log transformation or Poisson regression may be appropriate.
Power analysis requires estimates of variance and minimum detectable effect; pilot data can help.
Budget split should account for expected conversion rates and costs to ensure enough conversions per arm.
Test duration must cover full business cycles (e.g., weeks) to avoid day-of-week effects.
Follow-up analyses: pairwise comparisons with Bonferroni or FDR correction, segment analysis, and sequential testing for early stopping.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.