Snap·Technical Program Manager·Technical Phone Screen
- Walk me through the key ML evaluation concepts a TPM should know: recall, ROC curve, AUC, p-value, hypothesis testing, and Type I versus Type II errors.
- How would you explain AUC to a non-technical stakeholder?
- Offline AUC improved significantly in your model evaluation, but after launching the experiment, retention is dropping. What do you do?
- Design the end-to-end ML lifecycle for a content recommendation service at Snap, covering system components, stakeholders, launch metrics, and post-launch monitoring.
- Who are the key stakeholders for an ML-powered recommendation launch and how do you manage competing priorities across them?
- How would you diagnose model drift after a recommendation model has been in production for several weeks?
- How do you determine whether an experiment is underpowered before concluding it failed?
“I knew most of this but stumbled on explaining AUC in plain language.”