Meta·Data Scientist·Technical Phone Screen·Senior

SeniorPrefer not to say

Sep 2025Remote

Summary

Meta data scientist interview with a brutal production incident question about diagnosing a 10% drop in Likes on a global social network. Single question but it sprawled across like six sub-problems, and I kept second-guessing whether they wanted breadth or depth first.

Questions Asked (1)

Daily Likes on a global social network dropped 10% compared to the prior two-week average, but DAU held flat. Treat this as a live production incident. Walk through your full diagnostic plan: data validation checks and thresholds that point to instrumentation failure, how you'd rule out seasonality and external events, funnel and segment localization, three falsifiable hypotheses with the exact queries or plots to test them, a 24-hour decision tree to root cause, what you'd do if segments move in opposite directions, one immediate mitigation, and one follow-up experiment. Also name the first three queries or charts you'd run, and state the exact outcome that would tell you this is a behavior change rather than a logging issue.

Root Cause AnalysisProduct Analytics & MetricsA/B Testing & Experimentation

Author's notes

This one is basically a full take-home crammed into a live session.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by validating the data pipeline and metric definitions to rule out instrumentation failure, then systematically eliminate seasonality and external events before localizing the drop through funnel and segment analysis. Formulate falsifiable hypotheses with specific queries, follow a 24-hour decision tree to root cause, and prepare for conflicting segment movements with a mitigation and follow-up experiment.

Pro tip: Always check the simplest explanations first—like a logging bug or a holiday—before diving into complex behavioral theories. Document your queries and thresholds so you can quickly share findings and align with engineering and product teams.

1. Data Validation and Instrumentation Check

Verify data integrity by checking for logging errors, pipeline delays, or metric definition changes. Compare raw event counts and distinct user counts to expected ranges, and set thresholds like >5% deviation in logging volume or >2% drop in event-to-user ratio to flag instrumentation issues.

2. Rule Out Seasonality and External Events

Compare the drop to historical patterns (e.g., same period last year, day-of-week effects) and check for external events (holidays, outages, competitor launches). Use time-series decomposition or anomaly detection to see if the drop is within normal seasonal variation.

3. Funnel and Segment Localization

Break down the metric by funnel stages (e.g., like button impressions, clicks, confirmations) and segments (platform, region, user tenure, content type). Identify which specific segments or stages are driving the overall drop.

4. Hypothesis Testing and Decision Tree

Formulate three falsifiable hypotheses (e.g., UI bug, algorithm change, user behavior shift) and specify exact queries or plots to test each. Follow a 24-hour decision tree: if instrumentation issue, rollback; if behavior change, investigate further with experiments.

5. Mitigation and Follow-up

If segments move in opposite directions, prioritize the largest negative impact and implement an immediate mitigation (e.g., revert a recent change). Design a follow-up experiment to confirm root cause and prevent recurrence.

Key Points to Mention

First three queries/charts: 1) Time series of daily likes vs. DAU with confidence intervals, 2) Funnel conversion rates by stage, 3) Segment breakdown (e.g., by platform, region) of like rates.
Exact outcome indicating behavior change: If logging volume and event-to-user ratios remain stable, but like rates drop consistently across multiple independent data sources and segments, it's likely a behavior change.
Three falsifiable hypotheses: 1) A UI change reduced like button visibility (test: compare like rates for users exposed vs. not exposed), 2) An algorithm change reduced like prompts (test: A/B test or holdout analysis), 3) Seasonality (test: compare to historical same period).
24-hour decision tree: Hour 0-2: validate data; Hour 2-6: check seasonality/external; Hour 6-12: segment analysis; Hour 12-24: hypothesis testing and mitigation.
If segments move in opposite directions: isolate the net effect, check for Simpson's paradox, and prioritize the segment with the largest user impact or revenue impact.
Immediate mitigation: Roll back the most recent change if correlated with the drop. Follow-up experiment: A/B test to confirm the fix and measure long-term impact.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.