Openai·Machine Learning Engineer·Technical Phone Screen·Senior

Senior

Apr 2026

Summary

OpenAI ML engineer interview, one technical round focused entirely on implementing KV-cache generation for a transformer model. They gave you working code and asked you to make it efficient. Pretty deep dive for a single problem.

Questions Asked (1)

You're given a working transformer implementation with a basic greedy decode loop. Add KV-cache support so that autoregressive generation doesn't recompute attention over the full sequence on every step. The cache class is provided. Walk through all the changes needed and verify the outputs match the non-cached version exactly.

System DesignTechnical Trade-offsAlgorithms & Data Structures

Author's notes

This is the whole interview, basically.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by explaining the inefficiency of the greedy decode loop and how KV-cache eliminates redundant computation. Then walk through the necessary code changes: modifying the attention mechanism to use cached keys/values, updating the decode loop to pass and update the cache, and ensuring positional encodings are handled correctly. Finally, describe how to verify equivalence by comparing outputs and intermediate states.

Pro tip: Emphasize that the cache must be updated only with the new token's key/value at each step, and that positional encodings must reflect the absolute position to maintain correctness. Also, mention that you would test with varying sequence lengths and batch sizes to catch subtle bugs.

1. Understand the baseline and cache interface

Review the greedy decode loop and the provided cache class to understand its API (e.g., methods to update and retrieve cached keys/values). Identify where the full sequence is currently passed to attention.

2. Modify attention to accept and use cache

Change the attention function to accept cached keys/values, concatenate them with the new token's keys/values, and compute attention only over the cached plus new tokens. Ensure the cache is updated with the new keys/values.

3. Update the decode loop to manage cache

In the generation loop, initialize the cache, pass it to the model at each step, and update it with the new token's keys/values. Avoid recomputing keys/values for previously processed tokens.

4. Handle positional encodings correctly

Ensure that positional encodings are applied based on the absolute position of each token, not just the current step. This may require passing the current position index to the model.

5. Verify equivalence with non-cached version

Run both implementations on the same inputs and compare outputs token-by-token. Also compare intermediate attention outputs and logits to catch any discrepancies. Test with different sequence lengths and batch sizes.

Key Points to Mention

The KV-cache stores keys and values for all previous tokens, so attention only needs to compute for the new token.
The cache must be updated after each step with the new token's key and value.
Positional encodings must be applied correctly to the new token based on its absolute position in the sequence.
The attention mask must be adjusted to prevent attending to future tokens, but with cache, it's only needed for the new token against cached tokens.
Verification should include comparing logits and generated tokens, and testing with different batch sizes and sequence lengths.
Memory and computational trade-offs: KV-cache increases memory usage but reduces computation, enabling faster generation.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.