← Yahoo Interview Insights

Yahoo·Data Scientist·Technical Phone Screen·Senior

Senior
Sep 2025Remote

Summary

Yahoo DS interview with a meaty product analytics case about a DAU drop in Yahoo Mail. Single question but they clearly wanted to see the full investigation playbook, from data validation all the way through causal testing and alerting. Pretty intense for what felt like a mid-loop technical screen.

Questions Asked (1)

Q1

Yahoo Mail's DAU dropped 10% compared to the prior 7-day baseline. You're the on-call analyst. Walk through your full investigation: how do you verify the drop is real, isolate the cause using a metric tree and segmentation, run SQL or Python checks to find the top contributing segments, design a causal test to validate your hypothesis, and then recommend mitigations and a monitoring plan with rollback criteria. What do you do in the first two hours?

Root Cause AnalysisProduct Analytics & MetricsA/B Testing & Experimentation
Author's notes

This was a lot.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Structure your answer around a time-boxed incident response: first validate the drop is real and not a data artifact, then use a metric tree and segmentation to localize the cause, run quick SQL/Python checks to identify top contributing segments, and finally design a causal test to confirm the hypothesis before recommending mitigations and monitoring. Emphasize clear communication and prioritization within the first two hours.

Pro tip: Always check for data pipeline issues and metric definition changes first—many 'drops' are actually logging or ETL problems. Also, proactively set up a war room and communicate early with stakeholders to manage expectations and coordinate efforts.

1. Validate the Drop

Confirm the DAU drop is real by checking data freshness, pipeline health, and metric definitions. Compare with other sources (e.g., internal dashboards, logs) and rule out tracking outages or seasonality.

2. Localize with Metric Tree & Segmentation

Break DAU into components (new vs. returning, platform, geography, feature usage) and segment to find where the drop is concentrated. Use a metric tree to trace upstream drivers like login success rate, app opens, or email sends.

3. Run SQL/Python Checks for Top Segments

Write queries to quantify the contribution of each segment to the overall drop. Identify the top 2-3 segments accounting for the majority of the decline and check for correlated anomalies (e.g., error rates, latency).

4. Design a Causal Test

Formulate a hypothesis (e.g., a recent release caused crashes on Android) and design a test: compare pre/post, use holdout groups, or run an A/B test if feasible. Ensure the test isolates the suspected cause.

5. Recommend Mitigations & Monitoring

Propose immediate fixes (e.g., rollback, hotfix) and long-term monitoring with alert thresholds and rollback criteria. Define success metrics and a plan to track recovery.

Key Points to Mention

  • Check data quality and pipeline integrity before assuming a real user behavior change.
  • Use a metric tree to decompose DAU into actionable drivers (e.g., acquisition, retention, engagement).
  • Segment by dimensions like platform, geography, user cohort, and feature to isolate the drop.
  • Quantify segment contributions with SQL (e.g., window functions, group by) or Python (pandas).
  • Design a causal test (e.g., difference-in-differences, switchback) to validate the hypothesis.
  • Define rollback criteria and monitoring alerts to prevent future incidents.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.