Meta·Data Scientist·Technical Phone Screen·Senior

Senior

Jul 2026

Summary

Meta data scientist interview with a meaty stats/optimization question about threshold selection on a skewed score distribution. One question, lots of sub-parts, felt more like a take-home problem crammed into a live session.

Questions Asked (1)

You have a per-user interest score between 0 and 1 for a new feature, and the distribution is right-skewed. You can only contact at most K users per week. How would you pick a principled score threshold to maximize incremental value, given a per-contact cost and an expected benefit function? Walk through how you'd estimate the benefit function from historical data, find the profit-optimal cutoff, handle a left-skewed distribution, deal with sampling noise, and set up an ongoing backtest.

A/B Testing & ExperimentationProduct Analytics & MetricsTechnical Trade-offs

Author's notes

This one has five sub-parts and I fumbled the ordering badly.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Frame the problem as a constrained optimization: maximize total incremental profit (benefit minus cost) subject to a weekly contact limit K. Estimate the incremental benefit as a function of score from historical data (e.g., via causal inference or uplift modeling), then choose the threshold that maximizes profit while respecting K, and set up a continuous monitoring system to adapt to distribution shifts.

Pro tip: Always validate the benefit function with an A/B test or holdout before full deployment, and remember that the optimal threshold may be dynamic—re-estimate it periodically as user behavior and the score distribution evolve.

1. Define the objective and constraints

Clarify that the goal is to maximize incremental profit = expected benefit per contact − cost per contact, subject to contacting at most K users per week. Identify the decision variable: a score threshold τ such that users with score ≥ τ are contacted.

2. Estimate the incremental benefit function

Use historical data from past experiments or observational studies to model the incremental benefit as a function of the interest score. Techniques include uplift modeling, causal forests, or regression discontinuity if a threshold was previously used. Validate with a holdout set.

3. Find the profit-optimal cutoff

Given the estimated benefit function b(s) and cost c, compute the expected profit for each possible threshold τ: E[profit | τ] = ∫_{τ}^{1} (b(s) − c) f(s) ds, where f(s) is the score density. Choose τ* that maximizes this, but if the number of users above τ* exceeds K, select the top K users by score (or adjust τ to meet K).

4. Handle distribution skew and sampling noise

For right-skewed scores, the optimal threshold may be high; for left-skewed, it may be low. Use robust estimation (e.g., bootstrapping) to quantify uncertainty in b(s) and τ*. Consider Bayesian methods or regularization to avoid overfitting to noise.

5. Set up ongoing backtest and monitoring

Continuously track the actual incremental profit of contacted users versus a holdout group. Periodically re-estimate the benefit function and re-optimize the threshold. Use sequential testing or bandit algorithms to adapt in real time while maintaining statistical validity.

Key Points to Mention

Incremental value vs. total value: focus on the causal effect of contacting a user, not just correlation.
Constrained optimization: the threshold must respect the weekly contact limit K, which may require ranking users by score if the unconstrained optimal threshold selects more than K.
Benefit function estimation: use experimental data (A/B tests) or quasi-experimental methods (uplift modeling, causal inference) to estimate the incremental benefit as a function of score.
Distribution skew: right-skewed scores imply most users have low scores, so the optimal threshold may be high; left-skewed implies the opposite. The shape affects the density f(s) and thus the optimal cutoff.
Sampling noise: quantify uncertainty with confidence intervals or Bayesian credible intervals; use cross-validation or bootstrapping to avoid overfitting the threshold.
Ongoing backtest: implement a holdout group to measure true incremental lift, and use sequential testing or multi-armed bandits to adapt the threshold over time.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.