← Meta Interview Insights

Meta·Data Scientist·Technical Phone Screen·Senior

Senior
Jul 2026

Summary

Meta data scientist interview with a heavy stats focus. Four questions, all technical, no behavioral fluff. The kind of session where you either know your Bayesian priors or you don't.

Questions Asked (4)

Q1

Given an A/B test with 5 interim looks already done, compute the two-sided p-value and 95% confidence interval for the difference in conversion rates between control and treatment, then explain how O'Brien-Fleming alpha spending would affect whether you'd call the result significant.

A/B Testing & ExperimentationProduct Analytics & Metrics
Author's notes

This is the question I spent the most time on.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

First, clarify that with 5 interim looks, the naive p-value and CI are invalid due to inflated Type I error; compute them only for illustration, then apply O'Brien-Fleming alpha spending to determine the adjusted significance threshold. Finally, compare the observed test statistic to the adjusted boundary to decide if the result is significant at the final look.

Pro tip: Always emphasize that peeking at data without alpha adjustment is a common pitfall; mention that O'Brien-Fleming is conservative early and lenient later, which is ideal for Meta's iterative testing culture.

1. Compute naive test statistics

Calculate the two-sided p-value and 95% CI for the difference in conversion rates using standard formulas (e.g., two-proportion z-test), but note these are unadjusted and only for reference.

2. Explain alpha spending and O'Brien-Fleming

Describe how O'Brien-Fleming alpha spending divides the overall Type I error (α=0.05) across interim looks, with very small alpha at early looks and larger alpha near the end, preserving the overall error rate.

3. Determine adjusted significance boundary

Using the O'Brien-Fleming spending function, compute the adjusted critical z-value (or p-value threshold) for the final look (after 5 interim looks) and compare it to the observed test statistic.

4. Interpret significance and CI

If the observed test statistic exceeds the adjusted boundary, declare significance; otherwise, not significant. Also, note that the naive CI is too narrow; adjusted CIs can be computed but are often wider.

Key Points to Mention

  • Multiple testing inflates Type I error; naive p-value is not valid.
  • O'Brien-Fleming spending function: conservative early, less conservative later.
  • Alpha spending requires pre-specified number and timing of interim looks.
  • Adjusted p-value threshold at final look is smaller than 0.05 (e.g., ~0.04 for 5 looks).
  • Confidence intervals should also be adjusted for sequential testing.
  • Practical decision: if adjusted p < threshold, reject null; otherwise, continue or stop for futility.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.

Q2

You have five metrics with sorted p-values of 0.001, 0.012, 0.019, 0.070, and 0.300. Apply Benjamini-Hochberg at a 5% FDR and identify which are discoveries. Then compare to what Bonferroni would give you.

A/B Testing & ExperimentationProduct Analytics & Metrics
Author's notes

Went fine.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

First, explain the Benjamini-Hochberg procedure: rank the p-values, compute the BH critical values (i/m)*α, and find the largest i where p_i ≤ (i/m)*α. Then apply it to the given p-values to identify discoveries, and compare with Bonferroni by dividing α by the number of tests (0.05/5 = 0.01) and checking which p-values are below that threshold.

Pro tip: When comparing BH and Bonferroni, emphasize that BH controls FDR (expected proportion of false discoveries) while Bonferroni controls FWER (probability of any false discovery). In practice, BH is more powerful and better suited for exploratory analysis with many metrics, whereas Bonferroni is conservative and better when false positives are very costly.

1. Understand the goal and definitions

Clarify that we need to control the false discovery rate (FDR) at 5% using Benjamini-Hochberg, and compare with Bonferroni which controls the family-wise error rate (FWER).

2. Apply Benjamini-Hochberg procedure

Rank the p-values from smallest to largest: 0.001, 0.012, 0.019, 0.070, 0.300. For each rank i (1 to 5), compute the BH critical value (i/5)*0.05. Compare each p-value to its critical value and find the largest i where p_i ≤ (i/5)*0.05. All p-values up to that rank are discoveries.

3. Identify BH discoveries

Calculate critical values: i=1: 0.01, i=2: 0.02, i=3: 0.03, i=4: 0.04, i=5: 0.05. Compare: 0.001 ≤ 0.01 (yes), 0.012 ≤ 0.02 (yes), 0.019 ≤ 0.03 (yes), 0.070 ≤ 0.04 (no), 0.300 ≤ 0.05 (no). The largest i with a yes is i=3, so the first three p-values (0.001, 0.012, 0.019) are discoveries.

4. Apply Bonferroni correction

Bonferroni threshold is α/m = 0.05/5 = 0.01. Compare each p-value to 0.01: only 0.001 is below 0.01. So only the first metric is a discovery under Bonferroni.

5. Compare and interpret

BH finds 3 discoveries, Bonferroni finds 1. Discuss the trade-off: BH is less conservative and controls FDR, while Bonferroni controls FWER and is more stringent, reducing power.

Key Points to Mention

  • Benjamini-Hochberg procedure: rank p-values, compute (i/m)*α, find largest i with p_i ≤ (i/m)*α, reject all up to i.
  • Bonferroni correction: α/m threshold, controls family-wise error rate (FWER).
  • FDR vs FWER: FDR is the expected proportion of false discoveries among rejections; FWER is the probability of at least one false discovery.
  • BH is more powerful than Bonferroni, especially when many hypotheses are tested.
  • In A/B testing with multiple metrics, BH is often preferred for exploratory analysis to balance false discoveries and power.
  • The choice between BH and Bonferroni depends on the cost of false positives vs false negatives.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.

Q3

Your bot detection model has a 2% false positive rate and 10% false negative rate. Bot prevalence is 5%. If a user gets flagged, what's the actual probability they're a bot? Walk through the Bayes calculation.

Product Analytics & MetricsRoot Cause Analysis
Author's notes

Classic base rate trap and I almost fell into it.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

First, clearly define the events and given probabilities: P(Bot)=0.05, P(Flag|Bot)=0.10 (false negative rate means 10% of bots are missed, so true positive rate is 90%), and P(Flag|Human)=0.02. Then apply Bayes' theorem to compute P(Bot|Flag) = P(Flag|Bot)*P(Bot) / [P(Flag|Bot)*P(Bot) + P(Flag|Human)*P(Human)]. Finally, interpret the result in the context of bot detection, noting the impact of low prevalence.

Pro tip: Always sanity-check your result: with a 5% base rate, even a low false positive rate can lead to a surprisingly low precision, so emphasize that the model's flag alone is not definitive and additional evidence may be needed.

1. Define events and probabilities

Let B = user is a bot, F = user is flagged. Given: P(B)=0.05, P(F|B)=0.90 (since false negative rate is 10%), P(F|not B)=0.02. Also P(not B)=0.95.

2. State Bayes' theorem

Write P(B|F) = P(F|B)P(B) / P(F), where P(F) = P(F|B)P(B) + P(F|not B)P(not B).

3. Compute the denominator

Calculate P(F) = 0.90*0.05 + 0.02*0.95 = 0.045 + 0.019 = 0.064.

4. Compute the posterior probability

P(B|F) = (0.90*0.05) / 0.064 = 0.045 / 0.064 ≈ 0.703125, so about 70.3%.

5. Interpret the result

Explain that despite the low false positive rate, the low prevalence means that only about 70% of flagged users are actually bots, so 30% are false positives. Discuss implications for product decisions.

Key Points to Mention

  • Bayes' theorem and its application to classification metrics
  • Distinction between false positive rate and precision (positive predictive value)
  • Impact of base rate (prevalence) on posterior probability
  • Calculation of the denominator using law of total probability
  • Interpretation of the result in a business context (e.g., cost of false positives vs false negatives)
  • Potential next steps: adjusting threshold, gathering more signals, or using ensemble methods

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.

Q4

Average Order Value is right-skewed with a heavy tail. Compare IQR-based outlier removal, z-scores, and MAD or Huber M-estimators for handling this. Then explain a log-transform approach and how you'd report the effect back in original dollar terms using a smearing estimator.

Product Analytics & MetricsTechnical Trade-offs
Author's notes

Probably the most niche question.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by comparing the robustness and interpretability of IQR, z-scores, and MAD/Huber M-estimators for right-skewed AOV data, emphasizing that z-scores are inappropriate due to mean/sd sensitivity. Then explain the log-transform approach, including the need for a smearing estimator to correct bias when back-transforming to original dollars. Finally, discuss how to report the effect in dollar terms and validate the approach.

Pro tip: Mention that the smearing estimator is essentially the average of exponentiated residuals from the log model, and that ignoring it underestimates the mean on the original scale. Also, note that while log-transform handles skew, it may not fully address heavy tails if the distribution is extremely skewed, so consider robust regression or quantile regression as alternatives.

1. Compare outlier handling methods

Discuss IQR (non-parametric, uses quartiles, robust to skew), z-scores (parametric, assumes normality, sensitive to outliers), and MAD/Huber M-estimators (robust measures of scale and location, less affected by extreme values). Conclude that for right-skewed heavy-tailed AOV, IQR or MAD/Huber are preferable.

2. Explain log-transform approach

Describe taking the natural log of AOV to reduce skewness and make the distribution more symmetric, then performing analysis (e.g., regression) on log scale. Mention that this stabilizes variance and can make the data more normal-like.

3. Introduce smearing estimator

Explain that when back-transforming predictions or effects from log scale to original dollars, simply exponentiating gives the median, not the mean. The smearing estimator (Duan's smearing) corrects this by multiplying the exponentiated prediction by the average of exponentiated residuals.

4. Report effect in original dollars

Show how to compute the smearing factor: average of exp(residuals) from the log model. Then multiply the exponentiated predicted log value by this factor to get the expected AOV in dollars. Discuss interpretation and potential caveats.

5. Validate and consider alternatives

Mention the importance of validating the log-transform assumption (e.g., checking residual normality) and considering alternatives like GLMs with log link or quantile regression if the transform is inadequate.

Key Points to Mention

  • IQR is robust to skew and outliers, using median and quartiles.
  • Z-scores assume normality and are heavily influenced by outliers, making them unsuitable for heavy-tailed data.
  • MAD and Huber M-estimators provide robust measures of scale and location, less sensitive to extreme values.
  • Log-transform reduces skewness but requires smearing estimator for unbiased back-transformation to the mean.
  • Smearing estimator: average of exp(residuals) from the log model, multiplied by exp(prediction).
  • Alternatives: GLM with log link, quantile regression, or robust regression if log-transform is insufficient.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.