Blanked for a second because I wasn't expecting a pure stats question this early.
First, clarify that under the null hypothesis (no true effect), each sample's false positive rate is the significance level α (e.g., 0.05). Then, explain that the number of false positives across repeated samples follows a binomial distribution with parameters n=10,000 and p=α, and describe its shape, mean, and variance. Finally, discuss practical implications for A/B testing, such as expected false positives and multiple testing corrections.
Pro tip: Emphasize that the distribution is binomial, not normal, but for large n it approximates a normal distribution; also mention that this assumes independent samples and a fixed α. This shows you understand both the theory and its practical limits.
State that we assume the null hypothesis is true (no real effect) and each user is independent. Define the significance level α as the probability of a false positive per user.
Recognize that the number of false positives in a sample of 10,000 is a binomial random variable with n=10,000 and p=α. If α=0.05, the expected number is 500.
For large n, the binomial approximates a normal distribution with mean nα and variance nα(1-α). So the distribution is roughly symmetric and bell-shaped, centered at 500 with standard deviation ≈21.8 (for α=0.05).
Explain that across repeated samples, the number of false positives will vary around the mean, and this variability is captured by the standard deviation. This highlights the risk of false positives in A/B tests and the need for corrections like Bonferroni when testing multiple metrics.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.