Tubi·Data Scientist·Technical Phone Screen·Senior

Senior

May 2026

Summary

Tubi data science interview threw a deep stats question at me that was way more nuanced than I expected. The whole thing was basically a stress test on whether you actually understand p-values or just memorize the definition.

Questions Asked (1)

Explain to a non-technical product manager what a p-value of 0.03 from a two-sided A/B test actually means, and just as importantly, what it does NOT mean. Use a concrete scenario: baseline conversion 5.0%, observed lift of +0.4 percentage points, pooled standard error 0.18 pp. Cover the null hypothesis setup and the 'extremeness under the null' interpretation, why p is not the probability that the null is true, how p-values relate to confidence intervals and statistical power, the dangers of peeking at results early, and finish with both a plain-English explanation you'd give the PM and a Bayesian reframe of the same result.

A/B Testing & ExperimentationProduct Analytics & MetricsStakeholder Management

Author's notes

This question is a monster.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by grounding the explanation in the concrete numbers, then clearly define what the p-value is and is not, using the null hypothesis and extremeness. Connect it to confidence intervals and power, warn about peeking, and finish with a plain-English summary and a Bayesian reframe.

Pro tip: Use the concrete numbers to show that the observed lift is about 2.2 standard errors from zero, making the result statistically significant but not necessarily practically significant. Emphasize that the p-value does not measure the size or importance of the effect.

1. Set up the null hypothesis and observed result

State the null hypothesis (no difference in conversion rates) and the observed data: baseline 5.0%, lift +0.4 pp, pooled SE 0.18 pp. Compute the z-score: 0.4/0.18 ≈ 2.22.

2. Define the p-value correctly

Explain that p=0.03 means: if the null hypothesis were true, there is a 3% chance of observing a lift as extreme or more extreme than +0.4 pp (in either direction, since two-sided). It is not the probability that the null is true.

3. Relate to confidence intervals and power

Show that the 95% CI for the lift is approximately 0.4 ± 1.96*0.18 = [0.05, 0.75] pp, which excludes zero. Discuss how power affects the ability to detect a true effect and the risk of false negatives.

4. Warn about peeking and multiple testing

Explain that repeatedly checking results and stopping when p<0.05 inflates the false positive rate. Recommend fixed-horizon tests or sequential methods like alpha spending.

5. Plain-English summary and Bayesian reframe

Give a simple explanation: 'If there were truly no difference, we'd see a lift this big or bigger only 3% of the time.' Then reframe: with a skeptical prior, the posterior probability of a true effect might be much lower than 97%.

Key Points to Mention

Null hypothesis: no difference between control and treatment.
p-value is the probability of observing data as extreme or more extreme than observed, assuming the null is true.
p-value is not the probability that the null is true, nor the probability that results are due to chance.
Confidence interval: 95% CI for lift is roughly [0.05, 0.75] pp, indicating a positive effect but with uncertainty.
Statistical power: probability of detecting a true effect; low power increases false negatives.
Peeking inflates Type I error; use sequential testing or fixed sample sizes.
Bayesian reframe: posterior probability depends on prior; a skeptical prior may yield a much lower probability of a true effect than 1 - p.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.