TikTok·Data Scientist·Technical Phone Screen·Senior

Senior

Jun 2026

Summary

TikTok data scientist interview that went deep into causal ML territory. One big multi-part question covering the full pipeline from learner selection to A/B testing the final policy. Felt more like a research defense than a standard DS interview.

Questions Asked (1)

You have observational data from an always-on promotion and want to target users to maximize incremental revenue. Walk through a causal ML approach to estimate heterogeneous treatment effects and derive a targeting policy, covering: choice of meta-learner and why, model and cross-fitting setup, handling positivity violations, evaluation metrics including off-policy estimation, fairness constraints on the policy, and how you'd safely A/B test it before full rollout.

A/B Testing & ExperimentationTechnical Trade-offsProduct Analytics & Metrics

Author's notes

This was basically a dissertation prompt disguised as an interview question.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Structure your answer around the causal ML pipeline: start with problem framing and identification assumptions, then detail the meta-learner choice and cross-fitting setup, followed by handling positivity violations and evaluation via off-policy metrics, and finally address fairness constraints and a safe A/B testing plan. Emphasize trade-offs and practical considerations at each step, tying back to TikTok's scale and business context.

Pro tip: Show awareness that observational data from an always-on promotion likely suffers from confounding and positivity issues; propose sensitivity analyses and a phased rollout with guardrail metrics to build trust before full deployment.

1. Frame the causal problem and identification strategy

Define the treatment (promotion exposure), outcome (incremental revenue), and covariates. State assumptions (unconfoundedness, positivity, SUTVA) and discuss how you'd validate them, e.g., via domain knowledge or placebo tests.

2. Choose meta-learner and model setup

Select a meta-learner (e.g., X-learner or DR-learner) based on data imbalance and expected heterogeneity. Explain cross-fitting to avoid overfitting and use of flexible models (e.g., gradient boosting) for nuisance functions.

3. Handle positivity violations and estimate CATEs

Detect and address limited overlap via trimming, weighting, or targeted regularization. Estimate conditional average treatment effects (CATEs) and assess uncertainty, possibly using conformal prediction or bootstrap.

4. Derive targeting policy and evaluate with off-policy metrics

Construct a policy that targets users with positive predicted CATEs above a threshold. Evaluate using off-policy evaluation (e.g., IPS, DR) and metrics like Qini curve or policy value, comparing to baselines.

5. Incorporate fairness constraints and design safe A/B test

Define fairness constraints (e.g., demographic parity) and adjust policy via constrained optimization. Plan a phased A/B test with guardrail metrics, sequential testing, and a holdout to measure incremental revenue before full rollout.

Key Points to Mention

Choice of meta-learner (e.g., X-learner for imbalanced treatment groups, DR-learner for robustness) and rationale
Cross-fitting to prevent overfitting and enable valid inference
Positivity violations: detection via propensity score distribution, and remedies like trimming or overlap weights
Off-policy evaluation methods: IPS, DR, and metrics like Qini coefficient or policy value
Fairness constraints: definition, trade-offs with revenue, and constrained optimization techniques
A/B testing plan: phased rollout, guardrail metrics, sequential testing, and holdout groups

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.