This is the kind of question that sounds manageable until you're actually in it and realize how many directions it can go.
Start by acknowledging the gap between offline and production, then systematically isolate whether the issue is data, model, or infrastructure. Walk through a structured diagnosis: validate data consistency, check for distribution shift, examine serving logic, and finally consider model staleness or feedback loops.
Pro tip: Emphasize that you would first reproduce the issue in a staging environment with production-like data to avoid guesswork, and always quantify the impact (e.g., metric drop, business KPI) to prioritize fixes.
Clarify the performance drop: which metrics degraded, by how much, and when. Establish a baseline and impact on business KPIs.
Check for data drift, schema changes, missing values, or feature computation differences between offline and online serving.
Verify that the model artifact, version, and dependencies are consistent. Look for latency, throughput, or hardware issues affecting inference.
Compare production input distributions to training data. Consider concept drift, user behavior changes, or feedback loops from model predictions.
Apply targeted fixes (e.g., retrain, update features, adjust serving), then A/B test and set up continuous monitoring to prevent recurrence.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.