Upstart·Data Scientist·Technical Phone Screen
May 2026
A technical screen for a Data Scientist role at Upstart that was essentially one long, dense A/B testing case study. The question covered everything from data quality checks to the ship decision, and I spent most of the time wishing I had reviewed variance reduction techniques more recently.
- You're given results from an A/B test on a redesigned signup flow with columns for variant, sessions, signups, activation at day 7, p95 latency, support tickets, refund rate, and day-30 revenue. Walk through how you'd validate data quality, including sample ratio mismatch checks, bucketing integrity, and comparing exposure logs to analytics counts.
- How would you choose the primary metric and guardrail metrics for this experiment, and justify those choices given the available columns?
- How would you compute the treatment effect and confidence intervals for both simple proportion metrics and ratio metrics, and what would you do if the revenue distribution is heavily skewed?
- How would you assess whether the experiment ran long enough, and how does that relate to a pre-registered stopping rule and minimum detectable effect?
- How would you detect novelty or learning effects in this experiment, and what analyses would you run to separate them from a real treatment effect?
- Given that variant B shows better activation but also higher latency and more support tickets, walk through a concrete framework for making the ship or no-ship decision.
- Beyond the ship decision, what additional insights would you extract from this experiment data, such as segment-level heterogeneity, funnel step analysis, or form field sensitivity?
“This is where I started strong and then trailed off.” The rest of the author's notes on Data Scientist interview at Upstart, Technical Phone Screen round, covers how they worked through the question, what the panel pushed back on, and what they would do differently.
View Post