Meta·Data Scientist·Onsite - Product Sense / Strategy
Jun 2026
Integrity-focused DS case at Meta, all about designing a severity metric suite for harmful content. Heavy on measurement theory and tradeoffs, less coding. Walked out feeling like I'd spent an hour arguing with myself about what 'harm' even means.
- Design a metric or metric suite to measure the severity of harmful/policy-violating content on a large social platform. You have impression logs, content metadata, partial human-review labels, and ML classifier scores. What do you propose?
- The team wants to use View Prevalence (views of violating content divided by all views) as the primary KPI. What are the pros and cons?
- Walk through the key tradeoffs in choosing and optimizing these metrics. Cover at least: user safety vs. engagement, precision vs. recall, reporting robustness vs. sensitivity to change, and fairness across regions and languages.
- Recommend a final metric suite. Which metric is the primary KPI, what are your diagnostic metrics, and what are your guardrails?
- A new classifier improves recall by 15% in Spanish and Portuguese, and your severity-weighted metric rises as a result. How do you explain to leadership whether the platform actually got worse or your measurement improved?
- Would you use the weekly primary KPI to trigger on-call alerts for harm spikes, or would you design a separate alerting signal? What would that signal look like?
- How would you set severity weights across violation policy classes in a defensible way, and how would you detect if those weights are driving misleading movements in the KPI?
- An intervention reduces your primary metric but you suspect creators are migrating to a less-instrumented surface like DMs. How do you detect and account for surface displacement?
“This is the kind of question where jumping straight to a single ratio gets you killed.” The rest of the author's notes on Data Scientist interview at Meta, Onsite - Product Sense / Strategy round, covers how they worked through the question, what the panel pushed back on, and what they would do differently.
View Post