Start by reading the entire code and comments to understand the model architecture and locate the marked bugs. For each bug, explain why it's wrong and how to fix it, then verify the fix logically or with a quick test. Prioritize bugs that affect correctness of training or inference.
Pro tip: Demonstrate a systematic debugging process: after fixing each bug, articulate how you would test it (e.g., shape checks, gradient checks, or unit tests) to ensure the fix works and doesn't break other parts.
Read through the entire implementation and comments to grasp the transformer's components and identify where the 5 bugs are marked. Note the expected behavior of each part.
For each bug, determine why it's incorrect by comparing against transformer best practices (e.g., attention scaling, masking, layer norm placement). Explain the impact on model performance.
Write the corrected code for each bug, ensuring the fix aligns with standard transformer implementations. Consider edge cases and maintain numerical stability.
Describe how you would test each fix, such as checking tensor shapes, running a forward pass with dummy data, or verifying gradients. Mention any potential side effects.
Recap the bugs and fixes, and discuss any trade-offs or lessons learned about debugging transformer models. Highlight the importance of thorough testing.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Start by defining the key-value cache and its role in autoregressive decoding, then explain how it avoids redundant computation by storing past keys and values. Finally, quantify the speedup and discuss trade-offs like memory usage.
Pro tip: Mention that the KV cache shifts the bottleneck from compute to memory bandwidth, and that techniques like paged attention or quantization help manage the memory overhead.
Explain that in autoregressive models like GPT, tokens are generated one at a time, with each new token conditioned on all previous tokens.
Briefly outline how self-attention computes queries, keys, and values for each token, and how the output is a weighted sum of values based on query-key similarities.
State that the KV cache stores the key and value vectors for all previously generated tokens, so they don't need to be recomputed at each step.
Show that without caching, each step recomputes keys and values for all past tokens, leading to O(n^2) complexity; with caching, each step only computes for the new token, reducing to O(n) per step and O(n^2) overall but with much smaller constant factor.
Mention that the KV cache increases memory usage linearly with sequence length, and that techniques like paged attention, quantization, or sliding window attention mitigate this.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.