Structure your answer around the request lifecycle: ingress and routing, inference serving with GPU autoscaling, stateful dependencies, model rollout, and observability/SLOs. Emphasize trade-offs (latency vs cost, consistency vs availability) and how you'd handle spikes and partial failures with concrete mechanisms like queueing, circuit breakers, and multi-region failover.
Pro tip: Anchor every design choice to a measurable SLO (e.g., p99 latency < 2s, 99.9% availability) and explain how you'd validate it under load; this shows production maturity beyond just listing components.
Ask about traffic patterns (QPS, burstiness), latency/throughput targets, model size, and availability goals. Define SLOs (e.g., p99 latency, error rate) and budget for failures.
Choose a serving stack (e.g., Triton, vLLM, TGI) with GPU nodes; use a load balancer/API gateway for routing, request queuing, and batching. Discuss model parallelism and quantization for efficiency.
Implement GPU-aware autoscaling (e.g., KEDA, custom metrics) with warm pools to avoid cold starts. Deploy across regions with health checks, DNS failover, and data replication for stateful dependencies.
Use canary or blue-green rollouts with versioned model artifacts. Handle tokenizers and embedding caches as sidecars or shared services with versioning and invalidation strategies.
Instrument metrics (latency, throughput, GPU utilization), logs, and traces. Implement rate limiting and circuit breakers; use load shedding and graceful degradation during spikes or failures.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.