The math itself is straightforward once you remember the half-width is z times s over root n.
First, recognize that the CI half-width is proportional to 1/√n. Then, compute the scaling factor: √(100/10000) = √(1/100) = 1/10. So the new half-width is 10 * (1/10) = 1. Explain that this occurs because the standard error of the mean decreases with the square root of the sample size, reflecting the averaging of independent observations.
Pro tip: Emphasize that this scaling assumes independent observations and constant population standard deviation; in practice, with large samples, you might also mention the central limit theorem ensuring normality of the sample mean.
State that the confidence interval half-width is proportional to the standard error, which is σ/√n. Thus, the half-width scales as 1/√n.
Calculate the ratio of the new sample size to the old: 10000/100 = 100. The square root of this ratio is 10, so the new half-width is the old half-width divided by 10.
Apply the scaling: old half-width = 10, so new half-width = 10 / 10 = 1.
Discuss that increasing sample size reduces variability of the sample mean because more data provides more precise estimates. The square root scaling reflects the diminishing returns of adding more samples.
Mention that this holds if the sample standard deviation remains the same and observations are independent. Also note that for large n, the t-distribution approaches the normal, so the critical value (≈1.96) remains essentially unchanged.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Start by clarifying that narrowing the confidence interval (CI) without increasing sample size means reducing variance or increasing the signal-to-noise ratio. Then systematically present at least four methods, explaining the statistical mechanism and the practical trade-offs for each, ideally with examples relevant to A/B testing at Google.
Pro tip: Emphasize that these methods often involve trade-offs between statistical power, bias, and practical constraints, and that the best approach depends on the specific experiment and business context. Mention that some methods (like CUPED) are particularly effective in reducing variance without introducing bias, which is highly valued in tech companies.
Restate the question to ensure understanding: narrowing the CI without increasing sample size means reducing variance or increasing effect size. Acknowledge that sample size is fixed, so we must focus on other levers.
List at least four methods: (1) reduce variance through better experimental design (e.g., stratification, blocking, paired designs), (2) use variance reduction techniques like CUPED or regression adjustment, (3) increase effect size by targeting a more responsive subpopulation or using a more sensitive metric, (4) improve measurement precision (e.g., reduce noise in data collection), (5) use one-sided tests or adjust confidence level (though this changes coverage).
For each method, briefly explain how it narrows the CI: e.g., stratification removes between-strata variance, CUPED uses pre-experiment data to explain post-experiment variance, targeting increases the true effect size, better measurement reduces noise.
For each method, articulate the trade-offs: e.g., stratification requires known covariates and may complicate analysis; CUPED requires pre-period data and may introduce bias if not applied correctly; targeting reduces generalizability; one-sided tests increase Type I error risk; changing confidence level affects interpretation.
Summarize that the best approach depends on context, and often a combination of methods (e.g., CUPED + stratification) is used. Highlight that at Google, techniques like CUPED are standard for variance reduction.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
First, clarify that raising alpha increases Type I error, so any 'narrower' CI is an artifact of a higher false positive rate, not more precision. Then explain the conditions where this is misleading—multiple testing, peeking, or when stakeholders misinterpret the CI as a precision measure—and use a non-technical analogy like a smoke detector to convey the trade-off.
Pro tip: Emphasize that the real risk is not just statistical but organizational: a wider CI with a lower alpha protects against costly false launches, while a narrower CI from higher alpha can erode trust when results fail to replicate.
Explain that alpha is the false positive rate; raising it makes CIs narrower but increases the chance of declaring a false effect.
List scenarios: multiple comparisons without correction, optional stopping (peeking), and when the CI is interpreted as a precision measure rather than a decision threshold.
Compare to a smoke detector: lowering the threshold (raising alpha) makes it beep more often, but many beeps are false alarms—so you can't trust a 'quieter' alarm as evidence of fewer fires.
Explain that a narrower CI from higher alpha may lead to launching a harmful feature, wasting resources, or damaging user trust when the effect doesn't replicate.
Suggest keeping alpha low, using sequential testing or corrections, and focusing on effect sizes and practical significance rather than CI width alone.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.