Applied Intuition·Machine Learning Engineer·Onsite - System Design / Architecture
May 2026
Applied Intuition ML engineer interview, system design round focused entirely on KV caching for transformer inference. Pretty deep technically, they wanted actual tensor shapes and API signatures, not just hand-waving about attention.
- Design and implement a key-value caching system for autoregressive inference in a Transformer decoder. Include tensor shapes, a per-layer cache API that works across decoding steps and batched inputs, memory limit handling, and support for both greedy and beam search.
- Walk through the complexity analysis and expected speedup of KV caching compared to full recomputation at each decoding step.
- How would you test correctness of the KV cache implementation, particularly for edge cases like an end-of-sequence token appearing in the middle of a batch?
“This one took me a minute to scope properly.” The rest of the author's notes on Machine Learning Engineer interview at Applied Intuition, Onsite - System Design / Architecture round, covers how they worked through the question, what the panel pushed back on, and what they would do differently.
View Post