Meta·Data Scientist·Technical Phone Screen·Intermediate

Intermediate

Apr 2026

Summary

Meta DS interview with a stats-heavy question on comparing two models using a hypothesis test. Pretty much one focused problem but they wanted the full treatment, not just a gut answer.

Questions Asked (1)

Model A had 90 successes out of 100 users and Model B had 85 successes out of 100 users. Based only on this data, can you say Model A is better? Walk through the statistical test, your hypotheses, the math, and what the result actually means.

A/B Testing & ExperimentationProduct Analytics & Metrics

Author's notes

I jumped straight to 'yes A is better' before they even finished the question, which was obviously wrong.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by clarifying that the question asks whether the observed difference (90% vs 85%) is statistically significant, not just practically. Then walk through a two-proportion z-test: state hypotheses, compute the test statistic and p-value, and interpret the result in context. Finally, discuss limitations and what additional information would be needed to make a robust decision.

Pro tip: Emphasize that statistical significance does not imply practical significance; with large samples, even tiny differences can be significant, so always consider effect size and business impact.

1. Clarify the question and assumptions

Restate that we are comparing two independent proportions from binary outcomes. Assume random sampling, independence, and that the normal approximation is valid (np and n(1-p) > 5).

2. State hypotheses

Null hypothesis (H0): p_A = p_B (no difference). Alternative hypothesis (H1): p_A ≠ p_B (two-sided) or p_A > p_B (one-sided) if we expect A to be better.

3. Compute the test statistic

Calculate the pooled proportion: p_hat = (90+85)/(100+100) = 0.875. Standard error: SE = sqrt(p_hat*(1-p_hat)*(1/100 + 1/100)) ≈ 0.0468. z = (0.90 - 0.85) / SE ≈ 1.07.

4. Determine p-value and interpret

For a two-sided test, p-value ≈ 0.285. Since p > 0.05, we fail to reject H0. There is not enough evidence to conclude Model A is better than Model B.

5. Discuss limitations and next steps

Mention that the test assumes no confounding, and that with only 100 users per model, the study may be underpowered to detect small differences. Suggest computing confidence intervals and considering practical significance.

Key Points to Mention

Two-proportion z-test is appropriate for comparing independent proportions.
Pooled proportion is used under the null hypothesis of equal proportions.
The observed difference of 5 percentage points is not statistically significant (p ≈ 0.285).
Confidence interval for the difference: (-0.042, 0.142) includes zero, reinforcing no significant difference.
Statistical significance does not equal practical significance; consider effect size and business context.
Sample size and power: with n=100 per group, the test may lack power to detect small but meaningful differences.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.