Start by understanding the expected behavior of each component (forward pass, attention, masking, positional embeddings, logits, decoding) and then systematically debug each one using small test cases and shape checks. Prioritize bugs that cause incorrect generation (e.g., causal masking, positional embeddings) and validate fixes with a simple sequence generation task.
Pro tip: Use unit tests for each component with known inputs and outputs to isolate bugs quickly, and always check tensor shapes and dtypes as they often reveal subtle issues.
Review the miniGPT code to identify each component and its intended function. Write down the expected input/output shapes and values for a simple example.
Check the forward pass for correct tensor operations and shapes. Verify attention scores are computed correctly (QK^T / sqrt(d_k)) and softmax is applied over the correct dimension.
Ensure the causal mask is applied before softmax to prevent attending to future tokens. Verify positional embeddings are added correctly and not swapped or misaligned.
Check that logits are computed from the final hidden states (often via a linear layer) and that decoding uses the correct strategy (e.g., greedy, top-k) with proper handling of sequence generation.
After fixing individual components, run a simple generation task (e.g., predict next character in a sequence) to ensure the model produces coherent output. Compare with a known correct implementation if possible.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
The shape management here is where I fumbled.
Start by explaining the inefficiency of recomputing keys and values for all previous tokens at each decoding step, then describe how to cache K/V per layer and only compute for the new token. Emphasize that the cached and non-cached outputs must match exactly under the same seed, and outline how you would verify this.
Pro tip: Mention that the KV cache trades memory for speed, and discuss how to handle memory growth for long sequences (e.g., sliding window or paging). Also, note that numerical differences can arise from different computation orders, so use deterministic operations and compare with a tight tolerance.
Describe autoregressive decoding and why recomputing K/V for all tokens at each step is wasteful. State the goal: cache K/V per layer to avoid redundant computation.
Propose a per-layer cache (e.g., a list of tensors or a dictionary keyed by layer index) that stores keys and values for all past tokens. Mention that the cache grows with sequence length.
For each layer, compute K and V only for the new token, then concatenate them with the cached K/V. Use the full K/V for attention. Update the cache with the new K/V.
Under the same seed, the cached and non-cached versions must produce identical outputs. Use deterministic operations and compare logits or generated tokens with a small tolerance.
Mention memory vs. speed trade-off, and potential optimizations like cache eviction, quantization, or paged attention for long sequences.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.