Creditkarma·Machine Learning Engineer·Onsite - System Design / Architecture
May 2026
Onsite system design loop at Credit Karma for an ML Engineer role. The whole session revolved around one meaty multi-task ranking problem with a bunch of follow-ups layered on top. Technically pretty intense, lots of debugging-under-pressure vibes.
- You're training a multi-task ranking model that predicts click, application, conversion, and approval. Training loss goes NaN after a few hundred steps. Walk through how you'd debug it.
- Offline metrics look great but the model underperforms in the live experiment. How do you investigate the offline-online gap?
- Conversion and approval labels have a positive rate around 0.1%. What loss functions and optimization strategies make sense for such extreme imbalance, and how does label delay factor in separately?
- A high-importance feature is always a default constant at serving time but is correctly populated in the offline training data. What do you do, and how would you have caught this earlier?
- Mixed precision training is causing instability but you need it for throughput. How do you keep both?
- Conversion labels take 14 days to mature. How does that change your loss construction and your offline evaluation setup?
- How would you set up continuous monitoring to automatically catch training-serving skew before it ships?
“The thing that tripped me up initially was wanting to list every possible cause (gradient explosion, bad LR, loss weights, etc.) before actually thinking about what the timing tells you.”