Start by clearly stating the inputs needed for a two-proportion z-test power calculation, then walk through the math step-by-step using the given baseline (22%) and minimum detectable effect (5% relative lift = 23.1% absolute). Conclude by explicitly calling out the assumptions baked into the formula and how violations of those assumptions would affect the result in a real TikTok experimentation context.
Pro tip: Interviewers at TikTok love when candidates go beyond the formula and mention practical constraints like novelty effects, network interference from the social graph, or the need for variance reduction techniques (CUPED) — this signals you've run real experiments, not just solved textbook problems.
State the null hypothesis (no difference in completion rates) vs. the two-sided alternative, and explicitly list all inputs: baseline p1 = 0.22, minimum detectable effect (MDE) as 5% relative lift giving p2 = 0.22 × 1.05 = 0.231, alpha = 0.05 (z_α/2 = 1.96), and power = 0.80 (z_β = 0.842).
Use n = (z_α/2 + z_β)² × [p1(1−p1) + p2(1−p2)] / (p1 − p2)², plug in the values to get approximately n ≈ (1.96 + 0.842)² × [0.22×0.78 + 0.231×0.769] / (0.011)², and walk through the arithmetic to arrive at roughly 44,000–47,000 users per arm.
Explicitly list the assumptions: observations are independent and identically distributed (i.i.d.), the normal approximation to the binomial is valid (large n), equal variance is not assumed since we use pooled or unpooled proportions, and the test is truly two-sided with no peeking (fixed horizon).
Discuss how TikTok-specific factors can violate assumptions — network effects and social sharing can break i.i.d., novelty bias can inflate early lift, and high user heterogeneity (content creators vs. passive viewers) may require stratification or CUPED to reduce variance and lower required sample size.
Tie the calculation back to the original scenario: if the experiment was underpowered (sample size below the calculated threshold), the non-significant result is inconclusive, not evidence of no effect — recommend re-running with adequate sample size before making a product decision.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Okay this is where I started sweating a little.
Begin by explaining the CUPED mechanism intuitively — that it removes pre-existing variance correlated with the outcome to isolate the true treatment effect — then walk through the mathematical relationship between covariate R² and effective sample size gain. Ground your answer in a concrete TikTok-relevant example (e.g., using pre-period watch time to reduce variance in post-period engagement) to demonstrate applied understanding.
Pro tip: Mention that CUPED and ANCOVA are mathematically equivalent under certain conditions, and flag practical pitfalls like covariate leakage (using data from after treatment assignment) or non-stationarity in user behavior — this signals production-level experimentation maturity that separates senior candidates.
Describe CUPED as a control variate method that constructs an adjusted outcome Ỹ = Y - θ(X - E[X]), where X is a pre-period covariate and θ is chosen to minimize residual variance. Emphasize that this adjustment is unbiased because the covariate is independent of treatment assignment.
State the key formula: Var(Ỹ) = Var(Y) × (1 - R²), so if R² = 0.25, the adjusted variance is 75% of the original. This directly translates to needing fewer observations to achieve the same statistical power.
Explain that since the required sample size scales with variance, a 25% variance reduction means you need only 75% of the original sample size for equivalent power — roughly a 33% efficiency gain. Alternatively, holding sample size fixed, you gain meaningful power to detect smaller effect sizes.
Walk through the practical steps: collect pre-period metric data (e.g., 7-day pre-experiment watch time), compute θ via OLS regression of Y on X using historical or holdout data, apply the adjustment at analysis time, and run standard t-tests on the adjusted metric. Stress that covariate selection and the pre-period window length are key tuning decisions.
Acknowledge that CUPED benefits diminish if the covariate is weakly correlated, and flag risks like covariate leakage, user behavior non-stationarity, and the assumption that θ is estimated independently of the current experiment. Mention that for new users with no pre-period data, fallback strategies (e.g., stratification) are needed.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
This one surprised me because it's genuinely a judgment call and there's no clean right answer.
Treat this as a decision problem rather than a purely statistical one by first diagnosing *why* the result is non-significant before committing to any path. Structure your answer around the trade-offs of each option using business context, statistical power, and the cost of continued experimentation. Ultimately, defend a nuanced position that shows you can balance rigor with pragmatism in a fast-moving product environment like TikTok.
Pro tip: Interviewers at companies like TikTok want to see that you understand the difference between 'absence of evidence' and 'evidence of absence' — explicitly distinguishing these signals shows statistical maturity and will set you apart from candidates who treat a p-value > 0.05 as a simple stop signal.
Before choosing a path, determine whether the non-significance is due to insufficient sample size/power, high variance, a truly null effect, or a poorly designed experiment. Check observed effect size, confidence intervals, and whether the experiment ran long enough to account for novelty effects or seasonality.
Assess the stakes: what is the cost of a false negative (missing a real effect) versus a false positive (shipping a harmful change)? At TikTok's scale, even small effect sizes on engagement or retention have massive downstream impact, so the acceptable error threshold should be business-driven, not just statistical convention.
If a power analysis reveals the experiment was underpowered and the observed effect size is practically meaningful, continuing data collection is defensible — but only with a pre-registered stopping rule to avoid p-hacking. Warn against peeking repeatedly without correction, as this inflates Type I error.
A Bayesian approach with a utility-based stopping threshold is particularly well-suited when you need continuous monitoring, have strong priors from historical experiments, or want to quantify the probability of a meaningful effect rather than just reject a null. Explain that this reframes the question from 'is there an effect?' to 'what decision maximizes expected value given current evidence?'
Make a clear, justified recommendation — for example, switching to a Bayesian framework if the team needs to make a ship/no-ship decision quickly, or stopping if the confidence interval already excludes any practically significant effect. Acknowledge the trade-offs of your choice and propose next steps such as a follow-up experiment with better instrumentation.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Standard territory but still easy to mess up under pressure.
Begin by clearly defining the multiplicity problem and why it inflates Type I error rates when testing multiple hypotheses simultaneously. Then walk through a structured decision framework for choosing the appropriate correction method based on the relationship between metrics and the cost of false positives vs. false negatives. Conclude by tying your answer to practical trade-offs in a product experimentation context like TikTok's.
Pro tip: Demonstrate that you understand corrections aren't always the right answer — sometimes pre-registering a strict metric hierarchy (primary, secondary, guardrail) is more practical and defensible than applying Bonferroni, which can be overly conservative and kill real signals in high-velocity experimentation environments.
Explain that running m simultaneous hypothesis tests at significance level α means the family-wise error rate (FWER) inflates to 1-(1-α)^m, making spurious findings likely. Clarify whether you're concerned with FWER or False Discovery Rate (FDR) depending on the stakes of the decision.
Pre-specify one primary metric that drives the ship decision, then categorize secondary metrics as either supporting evidence or guardrails. This reduces the effective number of tests requiring correction and aligns stakeholders before results are seen.
Select Bonferroni for strict FWER control when outcomes are independent and false positives are very costly; use Holm-Bonferroni for a less conservative sequential approach. For correlated metrics or exploratory analysis with many outcomes, prefer Benjamini-Hochberg FDR control, which is more powerful and better suited to high-throughput experimentation.
Acknowledge that TikTok metrics like watch time, likes, shares, and retention are often correlated, which means Bonferroni is overly conservative. Methods like Westfall-Young permutation or multivariate testing can exploit correlation structure to improve power while still controlling error rates.
Articulate the power cost of each correction to stakeholders — stricter corrections require larger sample sizes or longer experiments. Document the chosen method and metric hierarchy in the experiment design doc before launch to prevent p-hacking or post-hoc rationalization.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Start by explaining the core problem — repeated peeking inflates Type I error because each look is an additional hypothesis test — then walk through how alpha-spending functions like O'Brien-Fleming solve this by allocating a fixed error budget across planned interim analyses. Ground your answer in practical implementation details relevant to TikTok's high-velocity experimentation environment, such as handling continuous data streams and making early stopping decisions.
Pro tip: Mention the trade-off between conservative early boundaries (O'Brien-Fleming preserves most alpha for later looks) versus more aggressive early stopping (Pocock boundaries), and signal awareness that in industry settings like TikTok, you often care as much about early stopping for futility as for efficacy — this shows you think beyond textbook theory.
Explain that each interim look at the data is effectively an additional hypothesis test, and without correction, the cumulative probability of a false positive grows well beyond the nominal alpha (e.g., 5 looks at α=0.05 can push true Type I error above 20%). Establish this as the motivation for a structured sequential testing framework.
Describe how an alpha-spending function α(t) allocates the total error budget (e.g., 0.05) across information fractions t ∈ [0,1], where t represents the proportion of planned sample size collected. The key constraint is that the cumulative alpha spent at any interim look never exceeds the total budget, ensuring the overall Type I error rate is controlled.
Detail that O'Brien-Fleming uses a very conservative spending function early on (spending very little alpha at early looks), requiring extreme z-scores to stop early, while preserving most of the alpha budget for the final analysis — this minimizes the power loss compared to a fixed-sample test. Contrast this with Pocock boundaries, which spend alpha more uniformly but reduce final-look power more significantly.
Discuss practical considerations for TikTok's scale: pre-specifying the number and timing of interim looks, using information-based boundaries when sample sizes are uncertain, and integrating futility stopping (e.g., beta-spending or conditional power thresholds) to kill losing experiments early and free up traffic. Mention tools like gsDesign or sequential packages in R/Python.
Acknowledge that sequential testing adds operational complexity — analysts must commit to the stopping rules pre-experiment and avoid ad-hoc looks outside the plan. Briefly mention modern alternatives like always-valid p-values (e-values, mixture sequential probability ratio tests) or Bayesian adaptive designs as complementary approaches for continuous monitoring pipelines.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.