TikTok·Data Scientist·Technical Phone Screen·Senior

Senior

Jun 2026

Summary

A deep technical screen for a DS role at TikTok that was basically one giant causal inference design problem. The question covered so much ground it felt less like an interview and more like a take-home compressed into 45 minutes.

Questions Asked (1)

Design an incrementality test from an advertiser's perspective to measure whether TikTok ads actually drive lift in conversions. Walk through your randomization unit and why, how you'd construct the holdout group, your primary KPIs and their formulas, sample size and duration math, how you'd handle cross-channel spillovers and auction effects, what guardrails and falsification checks you'd put in, and how you'd report heterogeneous lift without p-hacking.

A/B Testing & ExperimentationProduct Analytics & MetricsTechnical Trade-offs

Author's notes

This was basically seven questions welded into one.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Frame the answer as a ghost-ad or intent-to-treat experiment where the advertiser randomizes at the user level within their own first-party data, then measures incremental conversions against a true holdout. Walk through the design choices in the order asked, explicitly connecting each choice to causal validity and practical constraints like budget and platform attribution.

Pro tip: Emphasize that the advertiser, not TikTok, must own the randomization and holdout to avoid platform-side selection bias, and pre-register the analysis plan to prevent p-hacking when slicing heterogeneous effects.

1. Define the causal estimand and randomization unit

State that you want the average treatment effect on conversions for users who could be exposed to TikTok ads. Choose user-level randomization within the advertiser's CRM or logged-in user base because it avoids contamination from device sharing and enables cross-device measurement.

2. Construct the holdout and treatment groups

Randomly assign eligible users to a holdout that receives no TikTok ads (or a ghost ad) versus a treatment group that is eligible for TikTok ads. Ensure the holdout is large enough to detect the expected lift and that assignment is independent of user behavior.

3. Select primary KPIs and compute sample size/duration

Define incremental conversions as the difference in conversion rates between treatment and holdout, and incremental ROAS as incremental revenue divided by incremental ad spend. Use power analysis with baseline conversion rate, minimum detectable effect, power, and significance level to determine sample size, then translate to duration based on daily eligible traffic.

4. Address spillovers, auction effects, and guardrails

Mitigate cross-channel spillovers by isolating the holdout from other TikTok touchpoints and using intent-to-treat analysis. Monitor auction dynamics by checking if treatment group ad delivery affects holdout through budget pacing. Set guardrails on brand metrics, user experience, and cost per incremental conversion, and run falsification tests like A/A tests and pre-period placebo checks.

5. Report heterogeneous lift without p-hacking

Pre-register subgroup analyses (e.g., by demographics, past purchase behavior) and use techniques like CUPED to increase power. Report confidence intervals and effect sizes for pre-specified subgroups, and apply corrections like Benjamini-Hochberg for multiple comparisons to avoid false positives.

Key Points to Mention

User-level randomization within the advertiser's first-party data to avoid platform bias and enable cross-device measurement.
Ghost ads or PSA holdout to maintain blind measurement and prevent spillover from other channels.
Intent-to-treat (ITT) analysis as the primary estimand, with treatment-on-treated as secondary if compliance is high.
Power analysis using baseline conversion rate, MDE, alpha, and power; duration based on daily eligible traffic and expected effect size.
Pre-registration of analysis plan and subgroup hypotheses to prevent p-hacking; use of CUPED and multiple comparison corrections.
Guardrail metrics such as brand lift, user experience, and cost per incremental conversion, plus falsification checks like A/A tests and placebo pre-periods.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.