Anthropic·Software Engineer·Onsite - System Design / Architecture
Jun 2026
System design round at Anthropic for a software engineer role, focused entirely on building an LLM request batching layer in front of a GPU fleet. Pretty deep dive, went through batch formation, concurrency correctness, and multi-GPU scaling all in one session.
- Design the core batching entry point: a blocking call that accepts a list of inputs, groups concurrent requests into batches, and returns outputs in the same order. How do you structure the in-memory buffer and what triggers a batch to flush?
- How do you make the batching system thread-safe when hundreds of threads are calling the entry point simultaneously? What shared state exists and what protects it?
- Generalize the design from one GPU to G GPUs. Compare a coordinator-push model versus a worker-pull model and argue for one.
- If input lengths vary a lot, how would you batch by total token budget instead of request count, and what changes in the buffer and flush logic?
- One request in a batch causes the GPU call to run 10x slower than normal. How do you prevent that from repeatedly hurting other requests in future batches?
- How would you add request priority (like paid vs free tier) without starving low-priority requests?
- What metrics would you instrument on this system in production, and which would you set alerts on?
“This is the meat of the whole problem.” The rest of the author's notes on Software Engineer interview at Anthropic, Onsite - System Design / Architecture round, covers how they worked through the question, what the panel pushed back on, and what they would do differently.
View Post