TikTok·Data Scientist·Technical Phone Screen·Senior

Senior

Jul 2026

Summary

TikTok data scientist interview that went deep into causal inference territory. The main question was a sprawling DiD design problem covering staggered rollouts, estimator choice, inference, and reconciliation with PSM. Felt more like a methods exam than a conversation.

Questions Asked (1)

You're running a staggered rollout across 50 regions over roughly 10 weeks with weekly revenue per user as your outcome. Design a difference-in-differences analysis that holds up under heterogeneous treatment effects. Walk through when two-way fixed effects is biased, what it actually identifies, and how you'd implement a Callaway-Sant'Anna or Sun-Abraham estimator including group-time ATTs and aggregation weights. Also cover the event-study setup, pre-trend testing with a joint F-test, inference under few clusters, and how you'd reconcile your DiD results with a parallel PSM analysis.

A/B Testing & ExperimentationTechnical Trade-offsProduct Analytics & Metrics

Author's notes

This was basically six questions stapled together and I definitely felt the seams.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by framing the staggered rollout as a DiD design, then explain why TWFE fails under heterogeneous effects and what it actually identifies. Walk through implementing Callaway-Sant'Anna or Sun-Abraham, including group-time ATTs, aggregation, event-study, pre-trend testing, and inference. Finally, discuss reconciling with PSM and practical considerations for TikTok's 50 regions.

Pro tip: Emphasize that with 50 regions, cluster-robust SEs may be unreliable; use wild cluster bootstrap or randomization inference. Also, highlight that PSM and DiD answer different questions—PSM estimates ATT for treated regions, while DiD estimates ATT under parallel trends—so reconciliation requires careful interpretation.

1. Explain TWFE bias and identification

Describe how TWFE with staggered adoption and heterogeneous effects uses already-treated units as controls, leading to biased estimates. Clarify that TWFE identifies a variance-weighted average of treatment effects, which can be negative even if all ATTs are positive.

2. Implement Callaway-Sant'Anna or Sun-Abraham

Outline the steps: define groups by treatment timing, estimate group-time ATTs using not-yet-treated or never-treated as controls, then aggregate with appropriate weights (e.g., group-size weights) to get overall ATT and event-study estimates.

3. Set up event study and pre-trend testing

Create event-time indicators relative to treatment, plot coefficients, and test joint significance of pre-treatment coefficients using an F-test. Discuss sensitivity to binning and anticipation effects.

4. Address inference under few clusters

With 50 clusters, use wild cluster bootstrap or randomization inference to obtain valid p-values. Mention that standard cluster-robust SEs may over-reject.

5. Reconcile with PSM

Explain that PSM can be used to create a matched control group, then run DiD on matched sample. Discuss how PSM and DiD address different biases and how to interpret discrepancies.

Key Points to Mention

TWFE bias under heterogeneous treatment effects and negative weighting
Callaway-Sant'Anna group-time ATTs and aggregation (simple, group, calendar, event-study)
Sun-Abraham interaction-weighted estimator and its implementation
Event-study specification and joint F-test for pre-trends
Inference with few clusters: wild cluster bootstrap, randomization inference
Reconciling DiD and PSM: different estimands, sensitivity to parallel trends assumption

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.