Meta·Data Scientist·Technical Phone Screen·Senior

Senior

May 2026

Summary

Meta data scientist interview with a meaty modeling question about predicting engagement uplift for a new product feature. One question, but it had a lot of moving parts and they clearly wanted to see end-to-end ML thinking.

Questions Asked (1)

How would you build a model to predict whether a new 'Show similar products' button increases buyer engagement, covering feature selection, algorithm choice, sample size, evaluation, and how you'd translate results into a launch recommendation?

A/B Testing & ExperimentationProduct Analytics & MetricsTechnical Trade-offs

Author's notes

This is the kind of question where you can spiral if you try to answer everything at once.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by clarifying the business goal and defining a measurable engagement metric, then frame the problem as a causal inference question rather than pure prediction. Propose an A/B test as the gold standard, but also discuss how you'd build a predictive model to estimate treatment effects and guide decisions. Structure your answer around feature selection, algorithm choice, sample size calculation, evaluation metrics, and translating results into a launch recommendation.

Pro tip: Emphasize that you would use a model to predict the treatment effect (uplift modeling) rather than just engagement, and that you'd validate with an A/B test to avoid confounding. This shows you understand both prediction and causal inference, which is crucial for product decisions.

1. Define the problem and metrics

Clarify what 'buyer engagement' means (e.g., clicks, purchases, time spent) and define the target population and success criteria. Identify whether the goal is to predict engagement or to estimate the causal impact of the button.

2. Feature selection and data collection

Select features that capture user behavior, product characteristics, and context (e.g., past engagement, product similarity, user demographics). Consider potential confounders and ensure data quality.

3. Choose algorithm and design experiment

For prediction, consider models like logistic regression, tree-based models, or neural networks. For causal effect, use uplift modeling or design an A/B test. Discuss trade-offs between model complexity and interpretability.

4. Determine sample size and evaluation plan

Calculate required sample size based on expected effect size, power, and significance level. Define offline evaluation metrics (e.g., AUC, precision@k) and online metrics (e.g., click-through rate, conversion rate).

5. Translate results into launch recommendation

Interpret model outputs and experiment results to estimate the button's impact on engagement. Consider practical significance, confidence intervals, and potential risks. Recommend launch, iterate, or abandon based on evidence.

Key Points to Mention

Causal inference vs. prediction: emphasize that you need to estimate the causal effect of the button, not just predict engagement.
Uplift modeling or A/B testing: discuss how to measure incremental impact and avoid confounding.
Feature engineering: include user-product interaction features, historical behavior, and product similarity metrics.
Sample size calculation: mention power analysis, minimum detectable effect, and duration of experiment.
Evaluation metrics: both offline (model performance) and online (business metrics) evaluation.
Launch recommendation: consider statistical significance, practical significance, and potential long-term effects.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.