Meta·Data Scientist·Technical Phone Screen·Senior

Senior

May 2026

Summary

Meta data scientist interview where a stakeholder scenario was used to probe statistical fundamentals. The question was framed as a live experiment review which added some pressure I wasn't expecting.

Questions Asked (1)

Define p-value, statistical power, Type I error, and Type II error in the context of A/B testing. Then explain why testing multiple metrics or variants requires corrections like Bonferroni, and walk through a concrete example.

A/B Testing & ExperimentationProduct Analytics & Metrics

Author's notes

The definitions part was fine, I've explained these enough times that it's almost muscle memory.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by clearly defining each term in the context of A/B testing, emphasizing their relationships. Then explain the multiple comparisons problem and why corrections like Bonferroni are necessary. Finally, walk through a concrete example with multiple metrics or variants, showing the impact of correction.

Pro tip: Mention that while Bonferroni controls the family-wise error rate, it can be conservative; in practice, techniques like Benjamini-Hochberg (FDR) or sequential testing are often used at scale, but Bonferroni is a good starting point for strict control.

1. Define core statistical concepts

Define p-value, statistical power, Type I error, and Type II error in the context of A/B testing, linking them to hypothesis testing.

2. Explain multiple comparisons problem

Describe how testing multiple metrics or variants inflates the chance of false positives (Type I error) and why corrections are needed.

3. Introduce Bonferroni correction

Explain the Bonferroni correction: divide the significance level (α) by the number of tests (m) to control the family-wise error rate.

4. Walk through a concrete example

Provide a numerical example with, say, 3 metrics and α=0.05, showing the adjusted threshold and the impact on false positive rate.

5. Discuss trade-offs and alternatives

Mention that Bonferroni is conservative and may increase Type II errors; briefly note alternatives like FDR control or sequential testing.

Key Points to Mention

p-value: probability of observing data as extreme as the sample, assuming the null hypothesis is true.
Statistical power: probability of correctly rejecting a false null hypothesis (1 - Type II error rate).
Type I error: false positive, rejecting a true null hypothesis; Type II error: false negative, failing to reject a false null hypothesis.
Multiple comparisons inflate Type I error; Bonferroni adjusts α to α/m to control family-wise error rate.
Example: With 3 metrics and α=0.05, Bonferroni threshold is 0.0167; without correction, chance of at least one false positive is ~14.3%.
Bonferroni is conservative; alternatives like Benjamini-Hochberg (FDR) or sequential testing may be more powerful.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.