Meta·Data Scientist·Onsite - Product Sense / Strategy·Senior

SeniorPrefer not to say

May 2026Remote

Summary

This was a product-heavy data science interview at Meta that threw me into a full-scale feature build decision for a group call product. One question, but it was basically a whole case study compressed into a single prompt. Walked out unsure if I'd said enough or way too much.

Questions Asked (1)

You need to decide whether to build a group call feature without relying on existing product usage tables. What non-table data sources would you pull from, what signal would you extract from each, and how would you quantify and de-bias them? Then design a full experiment plan to validate demand, including a fake-door test, a waitlist, and a limited beta, with hypotheses, eligibility criteria, success metrics, guardrails, sample size math, and a timeline anchored to today. Also cover how you'd segment results, control for novelty effects, and handle ethics.

A/B Testing & ExperimentationProduct Sense & IdeationProduct Analytics & Metrics

Author's notes

This question is enormous and I did not fully appreciate that until I was about four minutes in and realized I'd only covered support tickets.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by identifying non-table data sources (e.g., surveys, interviews, support tickets, social media) and extracting demand signals, then quantify and de-bias them using triangulation and weighting. Next, design a phased experiment plan (fake-door, waitlist, limited beta) with clear hypotheses, metrics, and sample size calculations, ensuring ethical considerations and novelty controls. Finally, outline segmentation and analysis strategies to validate demand robustly.

Pro tip: Anchor your timeline to today's date and explicitly state assumptions (e.g., baseline conversion rates) to show rigor; acknowledge that fake-door tests may miss nuanced user feedback, so pair them with qualitative research.

1. Identify non-table data sources and signals

List sources like user surveys, interviews, support tickets, social media mentions, app store reviews, and competitor analysis. Extract signals such as frequency of feature requests, sentiment, and unmet needs.

2. Quantify and de-bias signals

Assign weights to sources based on reliability and representativeness, adjust for biases (e.g., vocal minority, selection bias) using techniques like stratified sampling or inverse probability weighting, and triangulate to estimate demand.

3. Design phased experiment plan

Define hypotheses for each phase: fake-door (demand), waitlist (intent), limited beta (engagement). Specify eligibility criteria, success metrics (e.g., click-through rate, waitlist conversion, retention), guardrails (e.g., user satisfaction), and sample size calculations with power analysis.

4. Plan execution and analysis

Create a timeline anchored to today, including durations for each phase. Detail segmentation (e.g., by demographics, usage), novelty effect controls (e.g., holdout groups, longitudinal tracking), and ethical considerations (e.g., informed consent, privacy).

Key Points to Mention

Use of qualitative and quantitative non-table sources (surveys, interviews, support tickets, social listening) to capture demand signals.
De-biasing techniques: weighting, triangulation, and correction for selection and response biases.
Fake-door test design: measure click-through on a 'coming soon' button, with clear success thresholds and guardrails against false positives.
Waitlist mechanics: capture emails, measure sign-up rate and referral, and analyze drop-off to gauge intent.
Limited beta: define eligibility (e.g., power users), measure engagement, retention, and NPS, with sample size based on minimum detectable effect.
Segmentation and novelty controls: analyze results by user cohorts, use holdout groups, and track behavior over time to separate novelty from true demand.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.