← Meta Interview Insights

Meta·Data Scientist·Technical Phone Screen·Senior

Senior
Jun 2026

Summary

A deep-dive product analytics case for a DS role at Meta, all centered on designing a measurement framework for a customer-support chatbot. Single question but it had five sub-parts and honestly felt more like a take-home crammed into a live session.

Questions Asked (1)

Q1

You own a B2C customer-support chatbot aimed at reducing agent handoffs. Design a complete success framework: define a primary KPI that balances containment and customer utility (including its formula, eligible population, attribution window, and how to handle abandoned chats and recontacts within 72 hours), at least three guardrail metrics with thresholds, an experiment design covering randomization unit and seasonality, the instrumentation and logging you need, and a quasi-experimental fallback if randomization is not feasible.

A/B Testing & ExperimentationProduct Analytics & MetricsSystem Design
Author's notes

This was basically five questions glued together and they expected a coherent answer, not five separate answers.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by defining a primary KPI that balances containment and customer utility, then specify its formula, eligible population, attribution window, and rules for abandoned chats and recontacts. Next, outline guardrail metrics with thresholds, design an experiment with proper randomization and seasonality controls, and detail instrumentation and logging. Finally, present a quasi-experimental fallback for when randomization isn't feasible.

Pro tip: Emphasize that the primary KPI should be a composite metric that penalizes false containment (e.g., recontacts within 72 hours) to avoid gaming containment. Also, mention that guardrails should include both customer experience and operational metrics to catch unintended consequences.

1. Define Primary KPI

Propose a composite metric like 'Successful Containment Rate' that combines containment with customer utility, ensuring it penalizes recontacts and abandoned chats. Specify the formula, eligible population (e.g., all chat sessions excluding spam), attribution window (e.g., 72 hours), and how to handle abandoned chats (e.g., exclude from numerator but include in denominator) and recontacts (e.g., count as failure).

2. Set Guardrail Metrics

Identify at least three guardrails such as customer satisfaction (CSAT) score, average handle time for escalated chats, and agent escalation rate. Set thresholds based on historical baselines or business requirements, and explain how you would monitor them during the experiment.

3. Design Experiment

Outline an A/B test with randomization at the user level to avoid contamination, and include seasonality controls by stratifying or using time-based randomization. Mention sample size calculation, duration, and how to handle network effects if any.

4. Instrumentation and Logging

Detail the events to log: chat start/end, user messages, bot responses, containment events, escalations, recontacts within 72 hours, and abandoned chats. Ensure logging includes timestamps, user IDs, session IDs, and experiment group assignment for accurate analysis.

5. Quasi-Experimental Fallback

If randomization isn't feasible, propose a quasi-experimental design like difference-in-differences using a holdout group or pre-post analysis with matched control groups. Discuss how to account for confounders and validate assumptions.

Key Points to Mention

  • Composite KPI formula: (Successful contained chats - recontacts within 72h) / (Total eligible chats - abandoned chats)
  • Eligible population: all user-initiated chats excluding spam and internal tests
  • Attribution window: 72 hours to capture recontacts, with sensitivity analysis for different windows
  • Guardrail thresholds: e.g., CSAT not drop by more than 2%, escalation rate not increase by 5%
  • Randomization unit: user-level to prevent contamination, with stratification by user tenure or region
  • Quasi-experimental methods: difference-in-differences, synthetic control, or propensity score matching

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.