← Meta Interview Insights

Meta·Data Scientist·Technical Phone Screen·Senior

SeniorPrefer not to say
May 2026

Summary

Meta DS interview focused entirely on ranking model evaluation for Instagram Shopping. The question was meaty and required knowing both offline metrics and how they connect to actual business outcomes. Not a vibe check round at all.

Questions Asked (1)

Q1

What offline and online metrics would you use to evaluate a product-ranking model for a shopping feed? How do precision and recall apply here, and how do you reconcile offline model metrics with on-site business KPIs like conversion?

Product Analytics & MetricsA/B Testing & ExperimentationTechnical Trade-offs
Author's notes

This one took me a second to get traction on.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by defining offline ranking metrics (e.g., NDCG, MAP, precision@k, recall@k) and online business KPIs (CTR, conversion rate, revenue per session). Explain how precision and recall trade off in ranking, then describe how to validate offline improvements with online A/B tests and reconcile discrepancies through deeper analysis.

Pro tip: Emphasize that offline metrics are proxies, not goals—always tie them to online business impact and be ready to explain why an offline win might not translate online (e.g., position bias, feedback loops).

1. Define Offline Ranking Metrics

List metrics like precision@k, recall@k, NDCG, MAP, and MRR to evaluate ranking quality against labeled relevance data. Explain how precision and recall apply: precision measures how many retrieved items are relevant, recall measures how many relevant items are retrieved.

2. Define Online Business KPIs

Identify online metrics such as click-through rate (CTR), conversion rate, add-to-cart rate, revenue per session, and user engagement. These reflect actual business impact and user behavior.

3. Explain Precision-Recall Trade-off in Ranking

Discuss how increasing recall (retrieving more relevant items) may lower precision (more irrelevant items shown), and vice versa. In ranking, the trade-off is managed by position and threshold; e.g., top-k precision vs. overall recall.

4. Reconcile Offline and Online Metrics

Describe how to validate offline improvements with online A/B tests. If offline metrics improve but online KPIs don't, investigate factors like position bias, presentation, or metric misalignment. Use online metrics as the ultimate arbiter.

5. Iterate and Align Metrics

Continuously refine offline metrics to better correlate with online KPIs. Consider using counterfactual or unbiased offline evaluation methods to bridge the gap.

Key Points to Mention

  • Precision@k and recall@k: precision focuses on top results, recall on coverage of relevant items.
  • NDCG and MAP account for position and relevance grading, better for ranking than simple precision/recall.
  • Online metrics: CTR, conversion rate, revenue per session, and user satisfaction (e.g., dwell time).
  • A/B testing is the gold standard to measure causal impact of ranking changes on business KPIs.
  • Offline-online gap can arise from position bias, feedback loops, and differences in user intent.
  • Use online metrics to guide model selection, but offline metrics for rapid iteration and debugging.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.