Google·Software Engineer·Technical Phone Screen·Intermediate

Intermediate

May 2026

Summary

Interviewed at Google, got hit with an A/B testing question that sounds straightforward until you actually try to answer it completely.

Questions Asked (1)

What are the common pitfalls of A/B testing?

A/B Testing & ExperimentationProduct Analytics & Metrics

Author's notes

I went in thinking I'd just list a few things like peeking at results early and calling it a day.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by acknowledging that A/B testing is powerful but prone to pitfalls, then structure your answer around the experiment lifecycle: design, execution, analysis, and interpretation. For each phase, highlight 1-2 common pitfalls and briefly explain how to avoid them, emphasizing statistical rigor and practical significance.

Pro tip: Demonstrate maturity by discussing pitfalls beyond statistics, such as organizational challenges (e.g., coordination, metric ownership) and the importance of pre-registration to avoid p-hacking. Mention that at Google, they often use sequential testing and holdout groups to mitigate some pitfalls.

1. Design Phase Pitfalls

Discuss issues like insufficient sample size, lack of randomization, and unclear hypotheses. Emphasize the need for power analysis and defining primary metrics upfront.

2. Execution Phase Pitfalls

Cover problems like sample ratio mismatch (SRM), instrumentation errors, and external validity threats (e.g., novelty effects, seasonality). Mention the importance of monitoring during the test.

3. Analysis Phase Pitfalls

Highlight multiple comparisons, peeking (early stopping), and misinterpretation of p-values. Stress the need for corrections (e.g., Bonferroni) and sequential testing methods.

4. Interpretation Phase Pitfalls

Address confusing statistical significance with practical significance, ignoring confidence intervals, and overgeneralizing results. Emphasize effect size and business impact.

5. Organizational and Process Pitfalls

Mention lack of pre-registration, stakeholder misalignment, and ignoring qualitative feedback. Suggest establishing clear experiment review processes.

Key Points to Mention

Sample size and power: underpowered tests lead to false negatives; overpowered tests waste resources.
Peeking and early stopping: inflates false positive rate; use sequential testing or alpha spending.
Multiple comparisons: testing many metrics increases false positives; apply corrections like Bonferroni or FDR.
Sample ratio mismatch (SRM): indicates randomization or instrumentation issues; always check.
Novelty and primacy effects: short-term spikes may not persist; run tests long enough.
Statistical vs. practical significance: a small effect may be statistically significant but not worth implementing.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.