HubSpot·Software Engineer·Onsite - System Design / Architecture
- Design a GPU-backed inference API for serving deep learning models. Walk through the API design (sync vs async), request schema and versioning, routing, batching strategy, model loading and cold-start handling, and multi-model hosting on shared GPUs.
- How would you handle autoscaling across heterogeneous GPU nodes, and what placement strategy would you use for routing inference requests?
- What observability would you build into this system? Specifically around latency, throughput, and GPU metrics.
- How would you implement rate limiting and multi-tenant isolation in this inference platform?
- How would you approach failure handling, including retries, timeouts, and canary or A/B rollouts for new model versions?
“This is a big one and I didn't pace myself well.”