Flatiron Health·Data Scientist·Technical Phone Screen
- You're launching a new recommendation module on a content platform. Define a primary success metric with an exact formula, including randomization unit, numerator, denominator, inclusion/exclusion criteria, and a 28-day attribution window. Also define at least two guardrail metrics with thresholds and explain the trade-offs.
- What randomization unit and bucketing strategy would you use to avoid contamination across surfaces and sessions? How would you detect and triage Sample Ratio Mismatch and event-loss issues, and what p-value threshold would you use for the SRM test?
- How would you handle novelty effects and seasonality in this experiment? Describe a ramp schedule, how you'd use pre-period covariates to reduce variance, and propose a minimum test duration rule. Also, how would you control for multiple comparisons across five planned segmentation cuts?
- Walk through your decision rubric for shipping, holding, or iterating based on metric results. Specifically, what do you do if the primary metric is flat but a guardrail metric regresses?
“This is where I spent most of my energy and still felt like I left things on the table.”