Start by clarifying requirements such as model size, latency, throughput, and traffic patterns. Then propose a system architecture that maximizes utilization of the single GPU through techniques like dynamic batching, model optimization, and request queuing, while addressing scalability and fault tolerance with cloud services.
Pro tip: Emphasize that the GPU is the bottleneck, so every design decision should be justified by how it improves GPU utilization or reduces latency. Also, discuss cost implications and the trade-off between latency and throughput.
Ask about expected request rate, latency SLA, model size, and whether the model can be optimized. This shapes the entire design.
Propose a request queue, dynamic batching, and a model server (e.g., Triton, TorchServe) that runs on the GPU. Include autoscaling for the CPU components but note the GPU is fixed.
Discuss model quantization, pruning, and compilation (e.g., TensorRT) to reduce memory and increase throughput. Consider multi-model serving if applicable.
Use cloud load balancers, health checks, and fallback mechanisms. If the single GPU fails, have a cold standby or degrade gracefully.
Set up monitoring for GPU utilization, latency, and error rates. Use metrics to adjust batching parameters and trigger alerts.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.