Start by clarifying requirements and constraints (latency SLO, throughput, model size, GPU types), then walk through the end-to-end request flow from API gateway to GPU inference, highlighting key design decisions like stateless vs stateful, batching, and failure handling. Emphasize trade-offs, especially what to skip given the tight latency budget, and justify each component's necessity.
Pro tip: Focus on the latency budget: for near real-time, avoid synchronous persistence and complex retry logic; instead, use idempotent request IDs and client-side retries with exponential backoff. Also, mention that dynamic batching must balance latency and throughput, often using a small max batch size and a short timeout (e.g., 10ms).
Ask about latency SLO (e.g., p99 < 200ms), throughput (100K concurrent), model size, GPU types, and whether requests are stateless. This sets the stage for design decisions.
Outline a stateless API gateway that authenticates, rate-limits, and routes requests to inference servers. Use a load balancer to distribute across GPU fleet, and consider a queue for buffering if needed.
Decide if requests need persistence: for real-time, skip synchronous DB writes; use in-memory queues or logs for debugging. Implement idempotency keys to avoid duplicate inference, and client-side retries with backoff for failures.
Describe how to batch requests on the GPU: collect requests within a short window (e.g., 10ms) up to a max batch size, then run inference. Ensure no duplicate inference by deduplicating based on request ID before batching.
Given latency budget, skip components like synchronous persistence, complex orchestration, or multi-region failover. Keep essential: load balancer, batching, idempotency, and monitoring.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.