TikTok·Data Scientist·Technical Phone Screen·Senior

Senior

Jun 2025Remote

Summary

TikTok data science interview with a single meaty causal inference question about a notification feature rollout. No randomization was possible, so the whole thing was about justifying observational methods under real-world constraints. Brutally specific.

Questions Asked (1)

Your team launched a notification feature without randomization due to marketing constraints. Design a causal study to estimate its effect on 7-day retention, comparing propensity score matching and difference-in-differences across multiple dimensions: treatment and control definitions, covariates for conditional ignorability, common support, pre-trend checks, staggered rollout, clustering strategy, sensitivity analysis, and how you'd communicate results to leadership.

A/B Testing & ExperimentationProduct Analytics & MetricsTechnical Trade-offs

Author's notes

This was a lot.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by clarifying the treatment and control groups and the assumptions needed for causal inference, then outline how you would implement both propensity score matching and difference-in-differences, addressing key challenges like common support, pre-trends, and staggered rollout. Finally, discuss how you would validate the results through sensitivity analyses and communicate them to leadership with appropriate caveats.

Pro tip: Emphasize that the choice between PSM and DiD depends on the specific context and data structure, and that combining them (e.g., using PSM to create a matched sample for DiD) can sometimes yield more robust estimates. Also, highlight the importance of pre-registering your analysis plan to avoid p-hacking and to build trust with stakeholders.

1. Define Treatment and Control Groups

Clearly specify who received the notification (treatment) and who did not (control), ensuring that the control group is comparable. Consider using a subset of users who were eligible but not exposed due to technical or timing reasons.

2. Choose Covariates for Conditional Ignorability

Select pre-treatment covariates that affect both the likelihood of receiving the notification and the outcome (7-day retention). These should include user demographics, past engagement metrics, and any other relevant behavioral variables.

3. Implement Propensity Score Matching (PSM)

Estimate propensity scores using logistic regression or machine learning, then match treated and control units with similar scores. Check common support by examining the overlap of propensity score distributions and trimming non-overlapping regions.

4. Implement Difference-in-Differences (DiD)

Identify a pre-treatment period and a post-treatment period, and compare changes in retention between treatment and control groups. For staggered rollout, use modern DiD methods (e.g., Callaway & Sant'Anna, Sun & Abraham) to account for heterogeneous treatment effects.

5. Validate and Communicate Results

Conduct pre-trend checks for DiD, sensitivity analyses for unobserved confounding (e.g., Rosenbaum bounds for PSM, placebo tests for DiD), and cluster standard errors at the appropriate level (e.g., user or region). Communicate findings to leadership with clear assumptions, limitations, and confidence intervals.

Key Points to Mention

Treatment and control definitions: Ensure control group is not contaminated by the treatment and that both groups are comparable pre-treatment.
Covariates for conditional ignorability: Include variables that predict both treatment assignment and outcome, and avoid post-treatment variables.
Common support: Check overlap of propensity scores and trim treated units without comparable controls to reduce bias.
Pre-trend checks: For DiD, verify that treatment and control groups had parallel trends before the intervention.
Staggered rollout: Use appropriate methods (e.g., Callaway & Sant'Anna) to handle staggered adoption and avoid bias from heterogeneous treatment effects.
Clustering strategy: Cluster standard errors at the level of treatment assignment (e.g., user, region) to account for correlation within clusters.
Sensitivity analysis: Assess robustness to unobserved confounding (e.g., Rosenbaum bounds, placebo tests) and report how conclusions might change.
Communication to leadership: Present results with clear assumptions, limitations, and actionable insights, avoiding overclaiming causality.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.