Anthropic·Software Engineer·Onsite - System Design / Architecture
May 2026
System design interview at Anthropic for a software engineering role, focused entirely on building a production-grade GPU inference platform for large ML models. One very long, very deep question that branched into like ten sub-topics. Felt more like a staff-level architecture review than a typical interview.
- What does the public-facing inference API look like? Walk through the request and response structure, how you handle tenant identity, model version selection, and idempotency. When would you add an async or job-based API, and when do you need streaming responses?
- Describe the core architecture end to end: API gateway, CPU-side validation and preprocessing, scheduler, dynamic batching, GPU workers, model registry, and control plane. How does a request actually flow through all of this?
- How do you independently scale the CPU-side components versus the GPU pool? What signals drive each, and why should they be decoupled?
- Diagnostic scenario: CPU utilization is low but GPUs are completely saturated. How do you confirm that's actually the bottleneck and what do you do about it, in order?
- How do you implement dynamic or continuous batching under latency deadlines? How do you ensure fairness across tenants and handle backpressure?
- How do you manage GPU memory across multiple models and tenants? Cover weight residency, KV cache sizing, quantization, tensor parallelism, and any isolation mechanisms.
- How does model versioning work in the registry? How do you support A/B routing, canary rollouts, and rollbacks?
- How do you autoscale across a heterogeneous GPU fleet with different hardware types, throughput curves, and a mix of on-demand and spot instances?
- How do you handle model loading and warmup latency, including lazy loading of fine-tuned adapters like LoRA?
- Walk through your approach to reliability, observability, capacity planning, cost controls, and security for this system. Cover retry semantics, latency breakdown metrics, per-tenant quotas, and tenant isolation.
“Started okay.” The rest of the author's notes on Software Engineer interview at Anthropic, Onsite - System Design / Architecture round, covers how they worked through the question, what the panel pushed back on, and what they would do differently.
View Post