← Meta Interview Insights

Meta·Data Scientist·Technical Phone Screen·Intermediate

Intermediate
Apr 2026

Summary

Probability and stats question for a Data Scientist role at Meta. The setup was a chatbot evaluation scenario and it covered independence, expected counts, and hypothesis testing all in one prompt. Pretty dense for a single question.

Questions Asked (1)

Q1

Given a chatbot where P(honest) = 0.7 and P(relevant) = 0.8, and assuming independence: what's the probability an answer is both honest and relevant? Out of 1,000 answers, how many would you expect to be neither? And how would you run a hypothesis test to compare the relevance rates of two different LLMs at a 0.05 significance level?

A/B Testing & ExperimentationProduct Analytics & Metrics
Author's notes

Three parts crammed into one question, which I did not expect.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by computing the joint probability using independence, then derive the expected count for neither. For the hypothesis test, clearly state the null and alternative hypotheses, choose a two-proportion z-test, and outline the steps including checking assumptions and interpreting the p-value.

Pro tip: When discussing hypothesis testing, emphasize the importance of considering practical significance and potential confounding factors, not just statistical significance. Also, mention that in real-world A/B tests, you often need to account for multiple comparisons and sequential testing.

1. Calculate joint probability

Since honest and relevant are independent, P(honest and relevant) = P(honest) * P(relevant) = 0.7 * 0.8 = 0.56.

2. Find probability of neither

P(neither) = 1 - P(honest or relevant) = 1 - (P(honest) + P(relevant) - P(both)) = 1 - (0.7 + 0.8 - 0.56) = 1 - 0.94 = 0.06. So, out of 1000 answers, expected neither = 0.06 * 1000 = 60.

3. Set up hypothesis test

Define null hypothesis H0: p1 = p2 (relevance rates are equal) and alternative H1: p1 ≠ p2 (two-tailed). Choose significance level α = 0.05.

4. Conduct two-proportion z-test

Calculate sample proportions, pooled proportion, standard error, and z-statistic. Find the p-value and compare to α. If p-value < 0.05, reject H0; otherwise, fail to reject H0.

5. Interpret results and consider practical significance

Discuss the conclusion in context, including effect size and confidence intervals. Mention any assumptions (e.g., random sampling, independence) and potential limitations.

Key Points to Mention

  • Independence assumption and its implications
  • Calculation of joint and complementary probabilities
  • Expected counts from probabilities
  • Null and alternative hypotheses for two-proportion test
  • Two-proportion z-test steps and assumptions
  • Interpretation of p-value and practical significance

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.