Datadog·Machine Learning Engineer·Technical Phone Screen
- Implement the forward pass of a Grouped-Query Attention module from scratch using only basic tensor operations, no high-level attention helpers.
- How much memory does GQA save versus standard multi-head attention during autoregressive decoding, and why is memory bandwidth the bottleneck at decode time rather than raw compute?
- How would you convert an existing multi-head attention checkpoint into GQA? Specifically, how do you initialize the G KV heads from the original H KV heads?
- Where do rotary position embeddings get applied in a GQA setup, and does the grouped structure change anything about that?
- Implement the incremental decode step for GQA: given a cached K and V of length t and one new token, produce the next output and update the cache. What shapes change?
“The core of the whole interview.”