Summary
TikTok data scientist interview that went deep into experimentation statistics. One big multi-part question covering basically every corner of A/B testing you can imagine. Left feeling like I'd either nailed it or completely embarrassed myself, still not sure which.
Questions Asked(5)
This part I actually felt okay about.
Suggested Approach
Start by clearly stating the inputs needed for a two-proportion z-test power calculation, then walk through the math step-by-step using the given baseline (22%) and minimum detectable effect (5% relative lift = 23.1% absolute). Conclude by explicitly calling out the assumptions baked into the formula and how violations of those assumptions would affect the result in a real TikTok experimentation context.
Define the Hypotheses and Parameters
State the null hypothesis (no difference in completion rates) vs. the two-sided alternative, and explicitly list all inputs: baseline p1 = 0.22, minimum detectable effect (MDE) as 5% relative lift giving p2 = 0.22 × 1.05 = 0.231, alpha = 0.05 (z_α/2 = 1.96), and power = 0.80 (z_β = 0.842).
Apply the Two-Proportion Sample Size Formula
Use n = (z_α/2 + z_β)² × [p1(1−p1) + p2(1−p2)] / (p1 − p2)², plug in the values to get approximately n ≈ (1.96 + 0.842)² × [0.22×0.78 + 0.231×0.769] / (0.011)², and walk through the arithmetic to arrive at roughly 44,000–47,000 users per arm.
State Core Statistical Assumptions
Explicitly list the assumptions: observations are independent and identically distributed (i.i.d.), the normal approximation to the binomial is valid (large n), equal variance is not assumed since we use pooled or unpooled proportions, and the test is truly two-sided with no peeking (fixed horizon).
Address Real-World Complications
Discuss how TikTok-specific factors can violate assumptions — network effects and social sharing can break i.i.d., novelty bias can inflate early lift, and high user heterogeneity (content creators vs. passive viewers) may require stratification or CUPED to reduce variance and lower required sample size.
Connect Back to the Non-Significant Result
Tie the calculation back to the original scenario: if the experiment was underpowered (sample size below the calculated threshold), the non-significant result is inconclusive, not evidence of no effect — recommend re-running with adequate sample size before making a product decision.
Key Points to Mention
Okay this is where I started sweating a little.
Suggested Approach
Begin by explaining the CUPED mechanism intuitively — that it removes pre-existing variance correlated with the outcome to isolate the true treatment effect — then walk through the mathematical relationship between covariate R² and effective sample size gain. Ground your answer in a concrete TikTok-relevant example (e.g., using pre-period watch time to reduce variance in post-period engagement) to demonstrate applied understanding.
Explain the CUPED Intuition
Describe CUPED as a control variate method that constructs an adjusted outcome Ỹ = Y - θ(X - E[X]), where X is a pre-period covariate and θ is chosen to minimize residual variance. Emphasize that this adjustment is unbiased because the covariate is independent of treatment assignment.
Quantify the Variance Reduction
State the key formula: Var(Ỹ) = Var(Y) × (1 - R²), so if R² = 0.25, the adjusted variance is 75% of the original. This directly translates to needing fewer observations to achieve the same statistical power.
Translate to Power / Sample Size Impact
Explain that since the required sample size scales with variance, a 25% variance reduction means you need only 75% of the original sample size for equivalent power — roughly a 33% efficiency gain. Alternatively, holding sample size fixed, you gain meaningful power to detect smaller effect sizes.
Describe the Implementation Process
Walk through the practical steps: collect pre-period metric data (e.g., 7-day pre-experiment watch time), compute θ via OLS regression of Y on X using historical or holdout data, apply the adjustment at analysis time, and run standard t-tests on the adjusted metric. Stress that covariate selection and the pre-period window length are key tuning decisions.
Address Trade-offs and Limitations
Acknowledge that CUPED benefits diminish if the covariate is weakly correlated, and flag risks like covariate leakage, user behavior non-stationarity, and the assumption that θ is estimated independently of the current experiment. Mention that for new users with no pre-period data, fallback strategies (e.g., stratification) are needed.
Key Points to Mention
This one surprised me because it's genuinely a judgment call and there's no clean right answer.
Suggested Approach
Treat this as a decision problem rather than a purely statistical one by first diagnosing *why* the result is non-significant before committing to any path. Structure your answer around the trade-offs of each option using business context, statistical power, and the cost of continued experimentation. Ultimately, defend a nuanced position that shows you can balance rigor with pragmatism in a fast-moving product environment like TikTok.
Diagnose the Non-Significant Result
Before choosing a path, determine whether the non-significance is due to insufficient sample size/power, high variance, a truly null effect, or a poorly designed experiment. Check observed effect size, confidence intervals, and whether the experiment ran long enough to account for novelty effects or seasonality.
Evaluate the Business Context and Cost of Error
Assess the stakes: what is the cost of a false negative (missing a real effect) versus a false positive (shipping a harmful change)? At TikTok's scale, even small effect sizes on engagement or retention have massive downstream impact, so the acceptable error threshold should be business-driven, not just statistical convention.
Assess the Case for Continuing Data Collection
If a power analysis reveals the experiment was underpowered and the observed effect size is practically meaningful, continuing data collection is defensible — but only with a pre-registered stopping rule to avoid p-hacking. Warn against peeking repeatedly without correction, as this inflates Type I error.
Evaluate the Bayesian / Utility-Based Alternative
A Bayesian approach with a utility-based stopping threshold is particularly well-suited when you need continuous monitoring, have strong priors from historical experiments, or want to quantify the probability of a meaningful effect rather than just reject a null. Explain that this reframes the question from 'is there an effect?' to 'what decision maximizes expected value given current evidence?'
Defend Your Recommendation with Trade-offs
Make a clear, justified recommendation — for example, switching to a Bayesian framework if the team needs to make a ship/no-ship decision quickly, or stopping if the confidence interval already excludes any practically significant effect. Acknowledge the trade-offs of your choice and propose next steps such as a follow-up experiment with better instrumentation.
Key Points to Mention
Standard territory but still easy to mess up under pressure.
Suggested Approach
Begin by clearly defining the multiplicity problem and why it inflates Type I error rates when testing multiple hypotheses simultaneously. Then walk through a structured decision framework for choosing the appropriate correction method based on the relationship between metrics and the cost of false positives vs. false negatives. Conclude by tying your answer to practical trade-offs in a product experimentation context like TikTok's.
Define the Problem
Explain that running m simultaneous hypothesis tests at significance level α means the family-wise error rate (FWER) inflates to 1-(1-α)^m, making spurious findings likely. Clarify whether you're concerned with FWER or False Discovery Rate (FDR) depending on the stakes of the decision.
Establish a Metric Hierarchy
Pre-specify one primary metric that drives the ship decision, then categorize secondary metrics as either supporting evidence or guardrails. This reduces the effective number of tests requiring correction and aligns stakeholders before results are seen.
Choose the Right Correction Method
Select Bonferroni for strict FWER control when outcomes are independent and false positives are very costly; use Holm-Bonferroni for a less conservative sequential approach. For correlated metrics or exploratory analysis with many outcomes, prefer Benjamini-Hochberg FDR control, which is more powerful and better suited to high-throughput experimentation.
Account for Metric Correlation
Acknowledge that TikTok metrics like watch time, likes, shares, and retention are often correlated, which means Bonferroni is overly conservative. Methods like Westfall-Young permutation or multivariate testing can exploit correlation structure to improve power while still controlling error rates.
Communicate Trade-offs and Document Decisions
Articulate the power cost of each correction to stakeholders — stricter corrections require larger sample sizes or longer experiments. Document the chosen method and metric hierarchy in the experiment design doc before launch to prevent p-hacking or post-hoc rationalization.
Key Points to Mention
Hardest part of the whole interview.
Suggested Approach
Start by explaining the core problem — repeated peeking inflates Type I error because each look is an additional hypothesis test — then walk through how alpha-spending functions like O'Brien-Fleming solve this by allocating a fixed error budget across planned interim analyses. Ground your answer in practical implementation details relevant to TikTok's high-velocity experimentation environment, such as handling continuous data streams and making early stopping decisions.
Define the Problem: Why Peeking Inflates Error
Explain that each interim look at the data is effectively an additional hypothesis test, and without correction, the cumulative probability of a false positive grows well beyond the nominal alpha (e.g., 5 looks at α=0.05 can push true Type I error above 20%). Establish this as the motivation for a structured sequential testing framework.
Introduce the Alpha-Spending Function Concept
Describe how an alpha-spending function α(t) allocates the total error budget (e.g., 0.05) across information fractions t ∈ [0,1], where t represents the proportion of planned sample size collected. The key constraint is that the cumulative alpha spent at any interim look never exceeds the total budget, ensuring the overall Type I error rate is controlled.
Explain O'Brien-Fleming Specifically
Detail that O'Brien-Fleming uses a very conservative spending function early on (spending very little alpha at early looks), requiring extreme z-scores to stop early, while preserving most of the alpha budget for the final analysis — this minimizes the power loss compared to a fixed-sample test. Contrast this with Pocock boundaries, which spend alpha more uniformly but reduce final-look power more significantly.
Address Implementation in a Streaming/Industry Context
Discuss practical considerations for TikTok's scale: pre-specifying the number and timing of interim looks, using information-based boundaries when sample sizes are uncertain, and integrating futility stopping (e.g., beta-spending or conditional power thresholds) to kill losing experiments early and free up traffic. Mention tools like gsDesign or sequential packages in R/Python.
Discuss Trade-offs and When to Use Alternatives
Acknowledge that sequential testing adds operational complexity — analysts must commit to the stopping rules pre-experiment and avoid ad-hoc looks outside the plan. Briefly mention modern alternatives like always-valid p-values (e-values, mixture sequential probability ratio tests) or Bayesian adaptive designs as complementary approaches for continuous monitoring pipelines.
Key Points to Mention
Discussion(5)
Sign in to join the discussion.
Not having the boundary values memorized is completely fine for a phone screen, the conceptual structure is what they're actually testing. The thing worth knowing cold is the shape intuition: O'Brien-Fleming boundaries are roughly proportional to 1/sqrt(t) where t is the information fraction (how much of your planned sample you've collected so far). That means the boundary at your first interim look, say at 25% of data, is approximately twice as stringent as your final boundary. In practice this translates to something like a z-score threshold of 4.something at the first look versus the usual 1.96 at the end. The spending function framing is just a way to formalize how you allocate your total alpha budget across looks, and O'Brien-Fleming is conservative early because most experiments that would stop early on a genuine effect will still be detectable later with more data, so you're not giving up much power by being strict upfront. Where people get caught is conflating the spending function with the boundary itself: the spending function tells you how much cumulative alpha you've 'used' at each look, and you back out the boundary from that. If they push you on actual numbers again, it's totally reasonable to say you'd use a package like gsDesign in R or rpact to compute them rather than doing it by hand.
Your math is right and the instinct to write out the formula was correct. The verbal explanation of CUPED almost always sounds muddier than it is because you end up saying 'covariate-adjusted term' and the interviewer is nodding but you can't tell if they're following. The formula anchors it: your adjusted outcome is Y minus theta times X, where theta is chosen to minimize variance of that difference, which ends up being Cov(Y,X) over Var(X), essentially the OLS coefficient. The residual variance of that adjusted metric is Var(Y) times (1 minus R²), so with R² of 0.25 you're running the test on a metric with 75% of the original variance. Since sample size requirements scale with variance, you need 25% fewer observations to hit the same power, which is roughly the 15% improvement in standard error you mentioned (sqrt of 0.75 is about 0.866, so your confidence interval shrinks by around 13-15%). For TikTok specifically I'd expect them to push on what you'd actually use as the pre-period covariate, because completion rate is noisy and users who are brand new have no pre-period data at all, so you'd need a fallback for cold-start users, maybe population mean imputation or a separate stratum. The conceptual point they're really probing is whether you understand that CUPED only helps when the covariate is actually predictive of the outcome in the test period, and a pre-period covariate measured on a different content distribution can have surprisingly low correlation with test-period behavior on a platform that moves as fast as TikTok does.
Your CLT answer was fine, actually. The normality assumption for a two-proportion z-test rests entirely on the sample being large enough that the binomial distribution of each proportion approximates normal, and that's exactly what CLT gives you. The more interesting justification, which might have landed better, is that with a baseline of 22% and the n you'd calculate here (somewhere around 14,000 per arm depending on your exact pooled variance), you're nowhere near the edge cases where CLT struggles. What trips people up is forgetting to state the pooled proportion explicitly: p_pool = (p1 + p2) / 2 under the null, which you use to compute the standard error rather than using each arm's variance separately. The equal allocation assumption is worth defending too because it's not just a convenience, it's actually optimal for minimizing total sample size when the two proportions are close, which they are here (0.22 vs 0.231). If TikTok's traffic splits aren't perfectly 50/50 due to holdout constraints or ramp schedules, your realized power drops, and you'd want to flag that upfront rather than discover it post-hoc.
Honestly the Bayesian pivot is a reasonable answer but the utility function question is the real interview. Vague answers about 'working with the product team' are fine as a starting point but the follow-up they were fishing for is probably something like: you'd anchor the utility to a concrete business outcome, say, estimated watch time revenue per percentage point of completion rate improvement, then set your stopping threshold at the point where the posterior expected value of shipping exceeds the expected cost of the change (engineering debt, potential downside on other metrics). The cost side is often what gets skipped. If the video length change degrades creator satisfaction metrics even slightly, that has a real cost at TikTok's scale, and a utility function that ignores it will push you to ship things you shouldn't.
Pre-registration is the move that actually matters most here and it sounds like you hit it. Bonferroni and BH are both correct answers but the hierarchy point you mentioned wanting to add is worth knowing for next time: you define a primary metric, maybe one or two secondary confirmatory metrics, and then a pile of exploratory metrics. The confirmatory ones get tested at your pre-specified alpha (with Bonferroni or BH correction across just those), and the exploratory ones are explicitly labeled as hypothesis-generating with no alpha control claimed. That framing sidesteps a lot of the correction debate because you're not really doing inference on the exploratory set at all.