Amazon·Data Scientist·Technical Phone Screen·Senior

Senior

Apr 2026

Summary

Amazon data scientist interview with a single brutal causal inference question about medication reminder rollouts. No randomization, staggered adoption across markets and channels, and they wanted the full treatment: identification strategy, bias handling, modern DID estimators, matching fallback, negative controls, spillovers, and power calculations. It was a lot.

Questions Asked (1)

You can't randomize who gets medication subscription reminders because the product launched in a staggered way across markets and channels (push, email, SMS). Outcomes are CSAT (1-5) and 4-week retention, and there's potential household spillover. Design a full observational causal study covering: identification strategy and regression spec with fixed effects and clustered SEs; treatment/risk set definitions to avoid immortal-time bias; pre-trend diagnostics and a modern DID estimator choice; a matching or weighting backup plan; two negative controls and a falsification test; how you'd handle spillovers, missing CSAT, and channel self-selection; and a power/MDE outline.

A/B Testing & ExperimentationProduct Analytics & MetricsTechnical Trade-offs

Author's notes

This was basically seven questions stapled together and handed to me like it was one thing.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Frame the staggered rollout as a quasi-experiment using a modern difference-in-differences design with not-yet-treated units as controls, explicitly addressing immortal-time bias via a risk-set definition. Then layer on robustness checks: pre-trends, negative controls, spillover adjustments, and a matching/weighting backup, while quantifying power for the smallest meaningful effect.

Pro tip: Lead with the identification threat—staggered adoption with heterogeneous effects—and name-drop Callaway & Sant'Anna or Sun & Abraham to show you know why TWFE fails. Then tie every design choice back to the two outcomes (CSAT and 4-week retention) and the household spillover, showing you can balance rigor with product relevance.

1. Define treatment, risk set, and estimand

Specify treatment as market-channel-time adoption of reminders, and define the risk set as users eligible at each time (avoiding immortal-time bias by aligning time zero with eligibility). State the estimand: ATT of reminders on CSAT and 4-week retention, allowing for household spillovers.

2. Choose identification strategy and regression spec

Use a modern DID estimator (e.g., Callaway & Sant'Anna) with not-yet-treated as controls, including unit and time fixed effects and clustering SEs at the market-channel level. For spillovers, consider a household-level exposure model or spatial/network DID.

3. Run diagnostics and falsification tests

Test pre-trends via event-study plots and joint F-tests; include two negative controls (e.g., unrelated health outcome, pre-period placebo treatment) and a falsification test (e.g., randomize treatment timing in placebo).

4. Backup plan: matching or weighting

If parallel trends fail, use propensity score matching or inverse probability weighting on pre-treatment covariates (demographics, channel usage, baseline health) to construct a comparable control group, then re-estimate DID.

5. Handle missing data, self-selection, and power

Address missing CSAT via multiple imputation or IPW; model channel self-selection with a Heckman selection or instrumental variable; outline power/MDE using simulation or formulas for clustered designs, targeting the smallest detectable effect on retention.

Key Points to Mention

Immortal-time bias: align time zero with eligibility and use a risk-set approach to avoid conditioning on future treatment.
Modern DID estimators (Callaway & Sant'Anna, Sun & Abraham) to handle staggered adoption and heterogeneous treatment effects, avoiding TWFE bias.
Clustered standard errors at the market-channel level to account for within-group correlation and serial correlation.
Negative controls: an outcome unaffected by reminders (e.g., unrelated preventive care) and a placebo treatment in a pre-period; falsification test via randomization inference.
Spillover handling: household-level exposure or network effects, possibly using a spatial DID or saturation design.
Power/MDE: account for clustering, staggered design, and spillovers; simulate to find MDE for 4-week retention and CSAT.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.