Reddit·Data Scientist·Technical Phone Screen·Senior

Senior

May 2026

Summary

Reddit data scientist interview with a brutal causal inference question that basically asked you to design a full synthetic control study from scratch, no A/B test allowed. The kind of question where you realize halfway through your answer that you've opened like six cans of worms simultaneously.

Questions Asked (1)

A high-impact feature can't be A/B tested due to policy or infrastructure constraints, but leadership needs a go/no-go decision. Walk through a complete causal inference analysis plan using synthetic control (or justify a different method). Cover how you'd build the donor pool, choose the pre-intervention window, define primary and guardrail metrics, select predictors and tune weights, run diagnostics and placebo tests, handle sensitivity analyses, analyze heterogeneity, and ultimately map results to a staged launch decision.

A/B Testing & ExperimentationProduct Analytics & MetricsTechnical Trade-offs

Author's notes

This question is basically nine questions stapled together.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by framing the problem as a causal inference challenge where randomization is infeasible, then justify synthetic control as the preferred method over alternatives like difference-in-differences or interrupted time series. Walk through each component of the analysis in a logical sequence, emphasizing robustness checks and decision-making criteria. Conclude by translating statistical results into a staged launch recommendation that balances risk and impact.

Pro tip: Emphasize that synthetic control is most credible when you have a long, stable pre-period and a donor pool of untreated units that are truly comparable; if not, be transparent about limitations and propose a complementary method like Bayesian structural time series. Also, tie your guardrail metrics directly to Reddit's core values (e.g., user trust, community health) to show product sense.

1. Define the causal question and choose the method

Clarify the intervention, outcome, and decision timeline. Justify synthetic control over alternatives (e.g., DiD, ITS) by noting its ability to construct a counterfactual from weighted donors, especially when parallel trends are questionable.

2. Construct the donor pool and pre-intervention window

Select a set of untreated units (e.g., similar subreddits, user segments) that are unaffected by the feature and have stable pre-period data. Choose a pre-intervention window long enough to capture seasonality and trends, but not so long that structural breaks occur.

3. Define metrics and predictors, then tune weights

Specify primary metrics (e.g., engagement, retention) and guardrail metrics (e.g., reports, churn). Select predictors (e.g., pre-period outcomes, covariates) that are predictive of post-intervention outcomes. Use optimization (e.g., nested optimization or ridge regression) to find donor weights that minimize pre-intervention fit error.

4. Run diagnostics, placebo tests, and sensitivity analyses

Assess pre-intervention fit via RMSPE and plot synthetic vs. actual. Conduct in-space and in-time placebo tests to gauge significance. Perform sensitivity analyses by varying donor pool, predictors, and pre-period length to check robustness.

5. Analyze heterogeneity and map to a staged launch decision

Explore effect heterogeneity across subgroups (e.g., by user tenure, region) to inform targeting. Translate the estimated effect and uncertainty into a go/no-go recommendation, proposing a staged rollout with clear success criteria and monitoring.

Key Points to Mention

Justification for synthetic control vs. other quasi-experimental methods (e.g., DiD, ITS, matching) based on assumptions and data structure.
Donor pool selection criteria: similarity, stability, and absence of spillover effects; consider using a holdout set for validation.
Pre-intervention window length trade-off: enough data for stable weights vs. avoiding contamination from other changes.
Primary vs. guardrail metrics: primary measures feature success, guardrails ensure no harm to user experience or platform health.
Diagnostics: pre-intervention fit (RMSPE), placebo tests (in-space and in-time), and sensitivity to model choices.
Decision framework: how to interpret effect size and uncertainty (e.g., Bayesian posterior) to recommend launch, iterate, or abandon.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.