Meta·Machine Learning Engineer·Onsite - System Design / Architecture·Senior

Senior

Apr 2026

Summary

Meta ML engineer interview with a meaty system design question about building an evaluation framework for ads ranking. Not a lot of context given upfront, so you have to drive the structure yourself.

Questions Asked (1)

Design an evaluation framework for ads ranking.

A/B Testing & ExperimentationProduct Analytics & MetricsSystem Design

Author's notes

This one is broader than it sounds.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by clarifying the goal of ads ranking (e.g., maximize revenue, CTR, or long-term value) and the constraints (latency, fairness, etc.). Then outline a multi-layered evaluation framework covering offline metrics, online A/B tests, and long-term holdback studies, ensuring alignment with business objectives.

Pro tip: Emphasize that no single metric is sufficient; combine short-term engagement metrics with long-term value and guardrail metrics to avoid optimizing for clicks at the expense of user experience or advertiser value.

1. Define Objectives and Constraints

Clarify the primary goal (e.g., maximize ad revenue, CTR, or conversions) and constraints (e.g., latency, fairness, user experience). Align with stakeholders on success criteria.

2. Select Offline Metrics

Choose offline metrics that correlate with online performance, such as AUC, log loss, calibration, and ranking metrics (NDCG, MAP). Use counterfactual evaluation to estimate online impact.

3. Design Online Experiments

Plan A/B tests with proper randomization, sample size, and duration. Define primary and guardrail metrics (e.g., CTR, revenue, user satisfaction, ad load). Consider interleaving experiments for faster iteration.

4. Incorporate Long-Term and Holistic Evaluation

Use holdback experiments, long-term studies, and causal inference to measure delayed effects (e.g., user retention, advertiser LTV). Monitor for feedback loops and ecosystem health.

5. Iterate and Refine

Continuously monitor metrics, analyze failures, and refine the framework. Use multi-armed bandits or sequential testing for adaptive experimentation.

Key Points to Mention

Offline metrics: AUC, log loss, calibration, NDCG, MAP, and counterfactual evaluation (e.g., inverse propensity scoring).
Online metrics: CTR, CVR, revenue, ad load, user engagement, and guardrail metrics (e.g., user satisfaction, latency).
A/B testing best practices: randomization unit, sample size calculation, statistical power, and avoiding pitfalls like peeking.
Long-term effects: holdback experiments, user retention, advertiser LTV, and ecosystem health.
Causal inference: techniques like propensity score matching, instrumental variables, and difference-in-differences.
Business alignment: tying metrics to company objectives (e.g., revenue, user growth) and considering trade-offs.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.