Went with a pooled two-proportion z-test which felt right.
Start by clearly stating the null and alternative hypotheses for a two-proportion z-test, then compute the pooled proportion and standard error to get the z-statistic and p-value. Finally, compare the p-value to alpha=0.05 and conclude whether there is sufficient evidence that Model A outperforms Model B.
Pro tip: Mention that this is a one-sided test in disguise: although the question says 'two-sided test', the conclusion about A outperforming B requires a directional interpretation. Clarify that the two-sided p-value tests for any difference, but to claim A > B you need to check the direction of the effect and possibly halve the p-value (or use a one-sided test).
Define null hypothesis H0: p_A = p_B (no difference in success rates) and alternative hypothesis H1: p_A ≠ p_B (two-sided). Optionally note that the directional alternative H1: p_A > p_B would be one-sided.
Select a two-proportion z-test for independent samples, as we are comparing two proportions from independent groups with large enough sample sizes (n=100 each).
Calculate the pooled proportion p_hat = (90+85)/(100+100) = 0.875. Then compute the standard error SE = sqrt(p_hat*(1-p_hat)*(1/100+1/100)) ≈ 0.0468. The z-statistic is (0.90 - 0.85)/SE ≈ 1.069.
For a two-sided test, p-value = 2 * P(Z > |z|) ≈ 2 * 0.1425 = 0.285. (Using standard normal table or software.)
Since p-value (0.285) > alpha (0.05), fail to reject the null hypothesis. There is not enough evidence to conclude that Model A outperforms Model B.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
The CI part was fine mechanically, unpooled SE this time, interval comes out something like (-0.03, 0.13).
Start by clearly stating the formula for the confidence interval of the difference between two proportions, then compute the interval using the provided sample data. After obtaining the interval, interpret it in terms of practical significance by discussing the magnitude of the difference and its relevance to business metrics, not just whether it excludes zero.
Pro tip: Always relate the confidence interval to the minimum detectable effect (MDE) or the business's threshold for meaningful impact; this shows you understand that statistical significance alone doesn't guarantee practical value.
Write down the formula for the confidence interval of the difference between two independent proportions: (pA - pB) ± z* sqrt( pA(1-pA)/nA + pB(1-pB)/nB ). Mention that this assumes large sample sizes and independent samples.
Plug in the sample proportions and sizes, calculate the standard error, and find the margin of error using the z-score for 95% confidence (1.96). Then compute the lower and upper bounds of the interval.
Check if the interval contains zero. If it does not, the difference is statistically significant at the 5% level. Explain what this means in terms of the null hypothesis.
Discuss whether the entire interval lies above or below a threshold that matters for the business. For example, if the lower bound is greater than the minimum detectable effect or a business-relevant lift, then the result has practical significance.
Mention other factors like sample size, duration of the experiment, novelty effects, and potential biases that could affect the practical implications. Emphasize that practical significance depends on the specific product and goals.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Blanked for a second on whether to use one-sided or two-sided z-values here.
Start by clarifying the test design (two-sided, equal allocation) and then walk through the sample size formula for comparing two proportions. Plug in the baseline rate (0.85), the target rate (0.90), alpha=0.05, and power=0.80, and compute the required per-arm sample size. Finally, sanity-check the result and mention practical considerations like multiple testing or sequential analysis.
Pro tip: Meta often runs many experiments simultaneously, so mention that you would adjust for multiple comparisons (e.g., Bonferroni) or use sequential testing to control false positives, which can increase sample size. Also, note that the formula assumes no interference and independent observations, which may not hold in social networks.
Confirm the test is two-sided with equal allocation, and that the metric is a proportion (e.g., conversion rate). State that you assume independent observations and no interference.
Identify p1=0.85, p2=0.90, alpha=0.05 (two-sided), power=0.80. Use the standard formula for two proportions: n = (Z_{α/2} + Z_β)^2 * (p1(1-p1) + p2(1-p2)) / (p2-p1)^2.
Z_{α/2}=1.96, Z_β=0.84. Compute pooled variance or use unpooled as above. Calculate numerator and denominator to get n per arm.
Perform the arithmetic: (1.96+0.84)^2 = 7.84; p1(1-p1)=0.1275, p2(1-p2)=0.09, sum=0.2175; difference=0.05, squared=0.0025. n = 7.84 * 0.2175 / 0.0025 ≈ 682. Round up to 683 per arm.
Verify the result is reasonable (e.g., ~680 per arm). Mention that real-world factors like multiple testing, sequential analysis, or non-compliance may require larger samples.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Pretty quick answer: adjusted alpha becomes 0.05/10 = 0.005.
First, clarify the setup: two models tested across 10 independent intents, likely with per-intent hypothesis tests. Then explain that Bonferroni correction adjusts the significance threshold to α/10 (e.g., 0.005 for α=0.05) to control the family-wise error rate. Finally, discuss whether the conclusions change by comparing the original p-values to the corrected threshold, and consider the trade-off between Type I error control and statistical power.
Pro tip: Mention that while Bonferroni is conservative, it's appropriate when false positives are costly; however, if the intents are truly independent and you're interested in each individually, you might use a less stringent method like Benjamini-Hochberg to control FDR, but be clear about the assumptions.
Confirm that there are 10 independent hypothesis tests (one per intent) and that the goal is to compare two models. Identify the significance level (e.g., α=0.05) and whether the tests are two-sided.
Divide the original α by the number of tests (10) to get the corrected threshold (e.g., 0.005). Explain that this controls the family-wise error rate at 5%.
Compare each intent's p-value to the corrected threshold. If a p-value was significant before but not after, the conclusion for that intent changes from significant to not significant.
Acknowledge that Bonferroni reduces power and may miss true effects. Mention alternatives like Holm-Bonferroni or Benjamini-Hochberg if controlling false discovery rate is more appropriate.
State whether the overall conclusion about which model is better changes. If some intents lose significance, the evidence for a consistent winner may weaken, but if the effect is strong, it may persist.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
This was the most open-ended part and I think I did okay.
Start by clarifying the decision context and data characteristics, then compare exact tests, z-tests, and Bayesian methods in terms of assumptions, robustness, and interpretability. Emphasize that the choice depends on sample size, variance assumptions, and the need for probabilistic statements, and always tie the reporting back to actionable metrics.
Pro tip: At Meta, where experiments run at massive scale, you'll rarely use exact tests for primary metrics, but you should know when they're appropriate for guardrail or rare-event metrics. Also, Bayesian methods are often used for decision-making under uncertainty, but you must clearly communicate the priors and decision criteria to stakeholders.
Identify the goal: is it a frequentist hypothesis test, or do you need probability statements to support a business decision? Consider the cost of false positives/negatives and the need for interpretability.
Check sample size, distribution, variance homogeneity, and independence. Exact tests are preferred for small samples or when distributional assumptions are violated; z-tests require large samples and known variance.
Use exact tests (e.g., Fisher's exact, binomial) for small samples or rare events; z-tests for large-sample proportions/means with known variance; Bayesian methods when you want to incorporate prior information and directly quantify uncertainty.
For exact tests: p-value, confidence interval, effect size. For z-tests: z-statistic, p-value, confidence interval. For Bayesian: posterior distribution, credible interval, probability of superiority, and expected loss.
Translate the statistical results into practical terms: lift, revenue impact, and decision recommendations. Highlight any caveats or sensitivity analyses.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.