Meta·Data Scientist·Technical Phone Screen·Intermediate

Intermediate

Jul 2026

Summary

Meta data scientist interview with a stats-heavy question about evaluating a marketing campaign's CTR. The whole thing was basically a hypothesis testing deep dive, which I expected but still found myself fumbling through parts of it.

Questions Asked (1)

You're told a marketing campaign has a 4.2% click-through rate. Leadership wants to know if that's good or bad. How do you formally set up the hypothesis test, choose the right statistical test, and figure out what additional data you need? Walk through computing a p-value and confidence interval, and explain both statistical and practical significance.

A/B Testing & ExperimentationProduct Analytics & Metrics

Author's notes

This question has a lot of layers and I didn't pace myself well.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by clarifying that a single CTR value is meaningless without a benchmark or comparison group, so you must define a null hypothesis (e.g., CTR equals a baseline or industry average) and an alternative. Then select the appropriate test (one-sample proportion z-test or two-proportion z-test if comparing to a control), compute the p-value and confidence interval, and finally interpret both statistical and practical significance in the context of business goals.

Pro tip: Always ask about the baseline or control group first—without it, you can't test anything. Also, mention that at Meta's scale, even tiny differences can be statistically significant, so practical significance (effect size and business impact) matters more.

1. Clarify the question and define hypotheses

Ask what the 4.2% is being compared to (e.g., historical baseline, control group, industry benchmark). Then state the null hypothesis (H0: CTR = baseline) and alternative (H1: CTR ≠ baseline or > baseline).

2. Choose the appropriate statistical test

If comparing to a known baseline proportion, use a one-sample z-test for proportions. If comparing two groups (e.g., campaign vs. control), use a two-proportion z-test. Ensure assumptions (independence, sample size) are met.

3. Identify needed data and compute test statistic

Collect sample size (n), number of clicks (x), and baseline proportion (p0). Compute the z-statistic: z = (p_hat - p0) / sqrt(p0*(1-p0)/n). Then find the p-value from the standard normal distribution.

4. Compute confidence interval

Calculate a 95% confidence interval for the true CTR using p_hat ± z* * sqrt(p_hat*(1-p_hat)/n). This gives a range of plausible values for the population CTR.

5. Interpret statistical and practical significance

Compare p-value to alpha (e.g., 0.05) to determine statistical significance. Then assess practical significance by considering effect size, business impact, and whether the difference is large enough to matter for decision-making.

Key Points to Mention

Null and alternative hypotheses must be clearly defined before testing.
Use a one-sample or two-proportion z-test depending on whether you have a control group.
Sample size and number of clicks are needed to compute the test statistic and confidence interval.
P-value indicates the probability of observing the data if the null hypothesis is true; it does not measure effect size.
Confidence interval provides a range of plausible values for the true CTR and conveys precision.
Statistical significance does not imply practical significance; consider business context and effect size.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.