Perplexity AI·Machine Learning Engineer·Technical Phone Screen·Senior

Senior

May 2026

Summary

Interviewed for an ML engineer role at Perplexity AI. One question but it was a meaty one about data lifecycle management, the kind where there's no clean answer and you can tell they're just watching how you think.

Questions Asked (1)

How would you decide when to discard stale data, retrain a model, and measure whether the updated model is actually better?

A/B Testing & ExperimentationTechnical Trade-offsProduct Analytics & Metrics

Author's notes

I went straight to staleness thresholds and drift detection, which felt right, but I didn't connect it back to business impact fast enough.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Frame the answer around a continuous monitoring and experimentation loop: first detect data drift or performance degradation, then decide on retraining based on cost-benefit and business impact, and finally validate the updated model through rigorous offline evaluation and online A/B testing. Emphasize that the decision to discard stale data and retrain should be driven by measurable signals and aligned with product goals.

Pro tip: Tie every decision to a clear business metric and quantify the trade-off between model freshness and operational cost; interviewers at Perplexity AI will value pragmatism and a strong experimentation mindset.

1. Monitor and Detect Drift

Continuously track data distribution shifts (e.g., PSI, KL divergence) and model performance metrics (e.g., accuracy, latency, user engagement) to identify when stale data or concept drift degrades performance.

2. Assess Retraining Need and Cost

Evaluate the impact of degradation on business KPIs and compare the cost of retraining (compute, engineering time) against the expected gain; set thresholds for retraining triggers.

3. Retrain with Fresh Data

Decide on the window of data to use (e.g., last N days) and retrain the model, ensuring proper data versioning and reproducibility.

4. Offline Evaluation and Validation

Test the new model on a holdout set from recent data, compare against the current model using relevant metrics, and check for regressions across slices.

5. Online A/B Test and Rollout

Deploy the new model in a controlled A/B test, measure impact on key product metrics with statistical significance, and decide whether to fully roll out or iterate.

Key Points to Mention

Data drift detection techniques (e.g., PSI, KL divergence, KS test) and monitoring dashboards
Retraining triggers: performance thresholds, time-based schedules, or event-driven (e.g., sudden drop)
Cost-benefit analysis: compute resources, engineering effort, and potential revenue impact
Offline evaluation metrics: accuracy, precision/recall, AUC, and business-specific metrics
Online experimentation: A/B testing, statistical power, guardrail metrics, and novelty effects
Feedback loops: incorporating user interactions and ensuring data quality for retraining

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.