Gusto·Data Scientist·Technical Phone Screen·Senior

Senior

May 2026

Summary

Gusto data science interview with a pretty involved A/B testing question that covers everything from p-values to multiple comparisons to a final ship decision. One question but it has a lot of layers, felt more like a take-home problem than a quick phone screen.

Questions Asked (1)

You ran an A/B/n experiment with one control and two treatment variants. Given a table of aggregated results (group, users, conversions), write Python to compute p-values for each variant vs control, provide 95% confidence intervals for the lift, explain how you'd handle multiple comparisons, and make a ship or no-ship recommendation with reasoning.

A/B Testing & ExperimentationProduct Analytics & MetricsTechnical Trade-offs

Author's notes

This one took me a while to fully unpack.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by clarifying the experiment design and assumptions, then write Python code to compute p-values and confidence intervals for each variant vs control. After that, address multiple comparisons using a correction method, and finally make a ship/no-ship recommendation based on statistical and practical significance.

Pro tip: Always consider the business impact and practical significance, not just statistical significance. A statistically significant lift with a tiny effect size might not be worth shipping.

1. Clarify assumptions and data

Confirm the data structure, metric type (e.g., conversion rate), and assumptions like independence and random assignment. Discuss whether to use a one-tailed or two-tailed test.

2. Compute p-values and confidence intervals

Write Python code using scipy.stats to perform two-proportion z-tests for each variant vs control, and calculate 95% confidence intervals for the absolute and relative lift.

3. Adjust for multiple comparisons

Apply a correction method like Bonferroni or Benjamini-Hochberg to control the family-wise error rate or false discovery rate, and explain the trade-offs.

4. Make a ship/no-ship recommendation

Interpret the results in the context of business goals, considering statistical significance, practical significance, and potential risks. Recommend shipping the best variant or continuing the experiment.

Key Points to Mention

Two-proportion z-test for conversion rates
Confidence intervals for lift (absolute and relative)
Multiple comparisons correction (Bonferroni, Holm-Bonferroni, or Benjamini-Hochberg)
Statistical power and sample size considerations
Practical significance vs statistical significance
Business impact and decision-making framework

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.