Together AI·Software Engineer·Onsite - System Design / Architecture
Apr 2026
System design round at Together AI for a software engineer role. The whole thing was a deep dive into building a GPU-aware pod scheduler from scratch, and they really pushed on every layer of it.
- Design an object-oriented, GPU-aware pod scheduler and cluster manager. Nodes track total GPUs and running pods, pods track GPU requirements. Implement APIs for adding/removing nodes and pods, scheduling a pod to a node with sufficient free GPUs, and querying utilization and listings.
- What placement strategy would you use for scheduling pods onto nodes, and how do you justify that choice? Walk through how your indexes get updated on every add, remove, schedule, and evict operation.
- How would you handle concurrency, specifically simultaneous pod scheduling requests and node additions? What guarantees do you provide around idempotency and failure handling?
- Write pseudocode for the schedule_pod function using your chosen placement strategy. Include time and space complexity analysis for each API.
- How would you handle edge cases like a pod requiring more GPUs than any single node has, or fragmentation where many small pods occupy a large node leaving no contiguous capacity for a bigger pod?
“This was the core question and it took the whole session.”