Attentive·Data Scientist·Technical Phone Screen·Senior

Senior

Jul 2026

Summary

Interviewed for a Data Scientist role at Attentive and got hit with a stats-heavy A/B testing question that required knowing your multiple comparisons theory cold. Not a casual conversation.

Questions Asked (1)

A new messaging feature is A/B tested across 30 companies independently, each with a 50/50 split and a significance threshold of 0.05. You see 2 significant positive results, 1 significant negative result, and the rest are not significant. What do you conclude, and how do you handle the multiple testing problem before deciding to roll out the feature?

A/B Testing & ExperimentationProduct Analytics & MetricsTechnical Trade-offs

Author's notes

This one stung a bit because I jumped straight to 'two uplifts, feature works' before catching myself.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by acknowledging that with 30 independent tests at α=0.05, we expect about 1.5 false positives by chance alone, so the 2 positive and 1 negative significant results are not strong evidence. Then explain how to correct for multiple comparisons (e.g., Bonferroni, Benjamini-Hochberg) and emphasize the need to look at effect sizes, confidence intervals, and practical significance before rolling out.

Pro tip: Don't just apply a correction blindly—consider whether the tests are truly independent and whether you care about controlling family-wise error rate (FWER) or false discovery rate (FDR). Also, check if the negative result is a real harm signal that might outweigh any small positive effects.

1. Quantify the multiple testing problem

Calculate the expected number of false positives: 30 tests × 0.05 = 1.5. Note that observing 2 positive and 1 negative significant results is roughly in line with chance, so none may be real.

2. Apply a multiple comparison correction

Choose an appropriate method: Bonferroni (control FWER) or Benjamini-Hochberg (control FDR). Adjust p-values or thresholds accordingly and re-evaluate which results remain significant.

3. Assess practical significance and consistency

For any results that survive correction, examine effect sizes, confidence intervals, and whether the direction and magnitude are consistent across companies. A single significant result after correction may still be spurious.

4. Investigate the negative result

Treat the significant negative result as a potential harm signal. Even if not significant after correction, it warrants investigation to avoid rolling out a feature that hurts some users.

5. Make a rollout decision

If no results survive correction, conclude there is no reliable evidence of an effect and recommend not rolling out based on this data. If some survive, consider a follow-up confirmatory test or a staged rollout with monitoring.

Key Points to Mention

Expected false positives under the null (30 × 0.05 = 1.5)
Bonferroni correction (α / number of tests = 0.05/30 ≈ 0.00167)
Benjamini-Hochberg procedure for controlling false discovery rate
Effect size and confidence intervals, not just p-values
Practical significance vs. statistical significance
Risk of multiple testing and the need for pre-registration or hierarchical testing

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.