This is where I spent most of my mental energy.
Start by stating that the two-proportion z-test is the standard choice for comparing binary outcomes in A/B tests, then derive it from the Bernoulli distribution using the Central Limit Theorem. Explain why it is preferable to a t-test on proportions, emphasizing the correct variance structure and the large-sample normal approximation. Finally, clearly state the conditions under which the normal approximation holds.
Pro tip: Mention that in practice, the z-test and t-test yield nearly identical results for large samples, but the z-test is theoretically correct because the variance of a Bernoulli is determined by its mean. Also, note that some practitioners use a t-test as a conservative approach when sample sizes are small, but it's not strictly necessary.
State that the two-proportion z-test is the correct significance test for comparing binary outcomes in an A/B test, as it directly models the Bernoulli distribution of each group.
Show that each observation is Bernoulli(p), the sum is Binomial(n, p), and by the Central Limit Theorem, the sample proportion is approximately normal for large n. Then construct the z-statistic using the difference in proportions and its standard error.
Highlight that a t-test assumes an unknown variance estimated from the data, but for Bernoulli data, the variance is a function of the mean (p(1-p)), so the z-test uses the correct variance without estimating an extra parameter. This makes the z-test more powerful and theoretically appropriate.
List the conditions: large sample sizes (typically n*p ≥ 10 and n*(1-p) ≥ 10 for each group), independent observations, and random assignment. Mention that for small samples or rare events, exact tests like Fisher's exact test may be needed.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Numerical computation question mid-interview is always a little stressful.
First, compute the sample proportions for control and variant, then calculate the difference. Use the two-proportion z-test to compute the z-statistic and corresponding p-value, and construct a 95% confidence interval for the difference using the standard error. Show all formulas and intermediate calculations clearly.
Pro tip: Always state your hypotheses and check assumptions (e.g., sample size large enough for normal approximation) before diving into calculations. This demonstrates rigor and helps catch potential issues.
Let p_control and p_variant be the true delivery rates. State the null hypothesis H0: p_control - p_variant = 0 and alternative H1: p_control - p_variant ≠ 0 (two-sided).
Compute p_hat_control = 920/1000 = 0.92, p_hat_variant = 880/1000 = 0.88, and the observed difference d = 0.92 - 0.88 = 0.04.
Under H0, use pooled proportion p_pool = (920+880)/(1000+1000) = 0.90. Standard error SE = sqrt(p_pool*(1-p_pool)*(1/1000 + 1/1000)) ≈ 0.013416. Z = d / SE ≈ 2.981.
For a two-sided test, p-value = 2 * P(Z > |2.981|) ≈ 2 * 0.00144 = 0.00288. To three significant figures, p-value = 0.00288.
Use unpooled SE for CI: SE_CI = sqrt(0.92*0.08/1000 + 0.88*0.12/1000) ≈ 0.013416. 95% CI = d ± 1.96*SE_CI = 0.04 ± 0.0263 = (0.0137, 0.0663).
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Fisher's exact was the right call and I knew it, small counts break the normal approximation.
Start by recognizing that with small sample sizes, the normal approximation for the z-test is unreliable, so you should choose between chi-square and Fisher's exact based on expected cell counts. Compute the exact p-value using Fisher's exact test (or explain how to compute it via hypergeometric probabilities) and discuss the trade-offs between the two tests.
Pro tip: Mention that Fisher's exact test is conservative but valid for small samples, while chi-square may be anti-conservative; also note that in practice, you might use a mid-p adjustment or simulation to balance power and validity.
Calculate the expected cell counts under the null hypothesis. With 20/25 and 12/18, the overall conversion rate is (20+12)/(25+18)=32/43≈0.744. Expected counts: control successes=25*0.744≈18.6, failures≈6.4; variant successes=18*0.744≈13.4, failures≈4.6. All expected counts are ≥5, but some are close, and total N=43 is small.
The z-test for proportions assumes normal approximation, which is questionable with small samples and expected counts near 5. Chi-square test also relies on the same approximation and may be inaccurate. Fisher's exact test does not rely on large-sample approximation and is exact for 2x2 tables.
Given the small sample size and the need for exact inference, Fisher's exact test is the most appropriate. Chi-square could be used but may yield inaccurate p-values; z-test is not recommended. If using chi-square, apply Yates' continuity correction, but still note its limitations.
For Fisher's exact test, fix the margins (25,18,20,12) and compute the probability of observing the given table or more extreme using the hypergeometric distribution. The p-value is the sum of probabilities of tables with the same margins that are as or more extreme than the observed. Alternatively, use statistical software (e.g., scipy.stats.fisher_exact) to get the p-value.
Report the p-value and discuss its interpretation. Note that Fisher's exact test is conservative (higher p-values) and may lack power. Mention that with small samples, confidence intervals are wide, and practical significance should be considered alongside statistical significance.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Conceptually I had this but my examples were a bit generic.
Start by clarifying the equivalence conditions: large sample sizes, moderate proportions, and independent observations. Then explain that the two-sample t-test on binary data approximates the z-test for proportions when the normal approximation to the binomial holds. Finally, detail at least two failure modes where the approximation breaks down, such as small sample sizes or extreme proportions.
Pro tip: Emphasize that in practice, the z-test for proportions and the t-test on 0/1 data yield nearly identical results when np and n(1-p) exceed 5 or 10, but the t-test is more robust to variance estimation issues. However, always check assumptions; for small samples or rare events, exact methods like Fisher's exact test are preferable.
Define the two-sample t-test for proportions (applied to 0/1 data) and the z-test for proportions, noting they both compare two independent proportions.
Explain that they produce essentially the same result when sample sizes are large enough for the normal approximation (e.g., np ≥ 5 and n(1-p) ≥ 5 for both groups) and proportions are not extreme.
When sample sizes are small, the t-test's reliance on the t-distribution and pooled variance may not approximate the normal well, leading to inaccurate p-values and confidence intervals.
When proportions are near 0 or 1, the normal approximation is poor, and the t-test can produce invalid results (e.g., negative variance estimates or p-values > 1).
Discuss other breakdowns: unequal sample sizes with unequal variances, dependence between observations, or when using the t-test without continuity correction.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.