OpenAI·Data Scientist·Technical Phone Screen
Apr 2026
This was a technical screen for a DS role at OpenAI, centered entirely on one meaty experiment design problem about a free-trial offer. No behavioral stuff, no SQL, just a long case that kept branching into harder sub-questions. Felt more like a product analytics take-home compressed into a live conversation.
- For a free-trial experiment, how would you define the primary metric, key secondary metrics, and at least one guardrail metric? Give concrete definitions including exact denominators.
- Should you use ITT or per-protocol analysis for this experiment, and how do you handle users who were assigned to treatment but never actually saw the offer?
- Retention is only defined for users who signed up. How do you estimate the treatment effect on retention without introducing post-treatment selection bias?
- What bugs or setup problems would you look for when reviewing Python analysis code for this experiment, without writing any code yourself?
- How would you make the final ship/iterate/kill decision for this free-trial feature, and what additional analyses would you run to connect the results to long-term business value?
“I went with signup rate as primary, defined as unique users who started a trial divided by unique eligible visitors assigned to each variant.” The rest of the author's notes on Data Scientist interview at OpenAI, Technical Phone Screen round, covers how they worked through the question, what the panel pushed back on, and what they would do differently.
View Post