Credit Karma·Machine Learning Engineer·Onsite - System Design / Architecture
- How would you design a serving architecture capable of handling hundreds of millions of users across thousands of models?
- At serving time, how do you route an incoming request to the correct model out of thousands of options?
- What strategies would you use to efficiently manage and serve thousands of models simultaneously?
- What are the tradeoffs between keeping models resident in memory versus loading them from disk on demand?
- How would you design a scalable model-loading strategy for a system with thousands of models?
- How do you differentiate handling between frequently used (hot) models and rarely used (cold) models in a serving system?
- How do you reason about the tradeoff between memory utilization and serving latency in a large-scale ML system?
“This was the opening question and it set the tone.”