Meta·Machine Learning Engineer·Technical Phone Screen·Senior

SeniorPrefer not to say

May 2026

Summary

Short Meta interview loop, just one question about model freshness. Not much context given so hard to say how it went.

Questions Asked (1)

How do you determine when a machine learning model needs to be retrained or refreshed?

Product Analytics & MetricsTechnical Trade-offsSystem Design

Author's notes

I went with the obvious signals first, like performance degradation on held-out data, then talked about data drift and concept drift.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by framing retraining as a decision driven by monitoring, business impact, and cost-benefit analysis. Then walk through a systematic process: define triggers, monitor them, evaluate retraining options, and validate before deployment. Emphasize that retraining is not always the answer—sometimes data quality fixes or feature updates suffice.

Pro tip: Tie retraining decisions to business metrics (e.g., revenue, engagement) rather than just model metrics, and always compare the cost of retraining against the cost of degraded performance. At Meta, this means quantifying the impact on user experience and ad performance.

1. Define performance baselines and business KPIs

Establish baseline model performance and link it to key business metrics (e.g., CTR, conversion, user engagement). This helps quantify the impact of model degradation.

2. Monitor for data and concept drift

Continuously track input data distributions (data drift) and the relationship between features and target (concept drift) using statistical tests and alerts.

3. Set up automated triggers and alerts

Define thresholds for performance metrics (e.g., AUC drop >5%) and drift metrics that trigger investigation. Automate alerts to relevant teams.

4. Evaluate retraining options and trade-offs

Assess whether to retrain from scratch, fine-tune, or update features. Consider costs (compute, engineering time) vs. benefits (improved accuracy, business impact).

5. Validate and deploy with A/B testing

Before full deployment, validate the retrained model offline and then run online A/B tests to measure real-world impact and guard against regressions.

Key Points to Mention

Data drift and concept drift detection using statistical methods (e.g., KL divergence, PSI)
Performance degradation thresholds tied to business metrics (e.g., revenue loss per day)
Cost-benefit analysis: retraining costs vs. cost of poor predictions
Automated monitoring and alerting systems (e.g., Prometheus, custom dashboards)
Online evaluation via A/B testing and shadow deployment
Incremental learning vs. full retraining trade-offs

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.