Start by clarifying the input shapes and the GQA configuration (number of query heads, key/value heads, head dimension). Then walk through the forward pass step by step: linear projections, reshaping, repeating K/V heads to match query heads, computing scaled dot-product attention, and finally the output projection. Emphasize tensor shapes at each step and explain why GQA reduces memory and compute compared to MHA.
Pro tip: Mention that you would use an efficient repeat_interleave or expand operation for the K/V heads instead of a naive loop, and note that this can be fused or optimized in production. Also, highlight that GQA is a trade-off between MHA and MQA, and Datadog might care about inference latency and memory savings.
Confirm the input tensor shape (batch_size, seq_len, d_model), the number of query heads (H), key/value heads (G), and head dimension (D). Ensure H is divisible by G.
Apply linear layers to project inputs to queries, keys, and values. Reshape Q to (B, H, L, D) and K, V to (B, G, L, D).
Repeat each K/V head H/G times along the head dimension to get (B, H, L, D). Use repeat_interleave or expand for efficiency.
Compute attention scores as Q @ K^T / sqrt(D), apply softmax over the last dimension, and multiply by V to get the output (B, H, L, D).
Concatenate heads and apply the output linear projection to get the final output (B, L, d_model).
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
I knew the bandwidth argument in theory but stumbled explaining it concisely.
Start by explaining the memory savings of GQA: it reduces the KV cache size by a factor equal to the ratio of query heads to KV heads (e.g., 8x for 32 query heads and 4 KV heads). Then clarify that during autoregressive decoding, each token generation requires loading the entire KV cache from memory, so the bottleneck is memory bandwidth (bytes moved per second) rather than compute (FLOPs), because the arithmetic intensity is very low (one token at a time).
Pro tip: Quantify the savings with a concrete example (e.g., Llama 2 70B: 8x reduction in KV cache) and mention that GQA is now standard in production LLMs like Llama 2/3 and Mistral because it directly addresses the memory bandwidth wall at decode time.
Briefly explain that standard multi-head attention (MHA) has one key/value head per query head, while grouped-query attention (GQA) shares key/value heads across groups of query heads, reducing the number of KV heads.
State that the KV cache size scales with the number of KV heads, so GQA reduces memory by the ratio of query heads to KV heads (e.g., 32:4 gives 8x savings). Mention that this saving is per token and grows with sequence length.
Describe that during decoding, the model generates one token at a time, and for each token it must load the entire KV cache from memory to compute attention. This makes the operation memory-bound because the compute per byte is very low.
Note that during prefill (processing the prompt) or training, many tokens are processed in parallel, increasing arithmetic intensity and making the operation compute-bound. This contrast highlights why the bottleneck shifts at decode time.
Summarize that GQA alleviates the memory bandwidth bottleneck by reducing the KV cache size, enabling larger batch sizes and longer contexts, which is critical for efficient inference in production systems.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Explain that converting MHA to GQA involves grouping the original H KV heads into G groups and initializing each group's KV head by averaging or selecting from the original heads. Emphasize that this reduces KV cache size while preserving most of the model's performance, and mention the trade-off between memory savings and potential accuracy loss.
Pro tip: Mention that you can also initialize the query heads by replicating or averaging the original query heads to match the new head count, and that fine-tuning after conversion is often necessary to recover performance.
Clarify that MHA has H query heads and H key/value heads, while GQA has H query heads but only G key/value heads, where G < H and H is divisible by G.
Partition the H original KV heads into G groups, each containing H/G heads. Typically, consecutive heads are grouped together.
For each group, initialize the new KV head by averaging the weights of the heads in that group. Alternatively, you could select one head (e.g., the first) as the representative.
Keep the query heads unchanged, or if the number of query heads also changes, replicate or average them accordingly. Ensure other parameters (e.g., layer norms, biases) are copied directly.
After conversion, fine-tune the model on a small amount of data to adapt the new KV heads and mitigate any performance drop.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Short answer: RoPE goes on Q and K after projection, before the attention scores, same as always.
Clarify that rotary position embeddings (RoPE) are applied to the query and key vectors after the linear projections, before the attention score computation. Explain that in GQA, the grouped structure does not change the application point or method; RoPE is still applied per head to Q and K, but the key heads are shared across query groups, so the same rotated key is used for multiple queries. Emphasize that the grouping affects the number of key heads, not the positional encoding mechanism.
Pro tip: Mention that RoPE is applied after the linear projection and before the dot product, and that in GQA, since key heads are shared, the rotation is applied once per key head and reused, which is efficient. Also note that some implementations apply RoPE to the query and key projections separately, but the grouping doesn't alter the per-head rotation.
State that RoPE encodes position by rotating query and key vectors in 2D subspaces, typically applied after the linear projection and before attention scores.
Explain that in multi-head attention, RoPE is applied to each query and key head independently, after the linear transformation.
Define GQA: multiple query heads share a single key/value head, reducing KV cache size. Clarify that the number of key heads is smaller than query heads.
Confirm that RoPE is still applied to each query head and each key head after projection. Since key heads are shared, the same rotated key is used for multiple query heads.
Conclude that the grouped structure does not change the application of RoPE; it only affects how many key heads exist and how they are shared. The rotation is per head and independent of grouping.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
First, clarify the GQA setup: number of query heads, key/value heads, and head dimension. Then, walk through the incremental decode step: compute the new query, key, and value for the single token, append the new key and value to the cache, and compute attention output using the updated cache. Finally, explicitly state the shape changes: the cache grows along the sequence dimension, and the output shape remains (batch_size, 1, num_query_heads, head_dim).
Pro tip: Emphasize that GQA reduces KV cache size by sharing key/value heads across groups of query heads, so the cache shape uses num_kv_heads instead of num_query_heads. Mention that the incremental step avoids recomputing past keys/values, which is crucial for efficient autoregressive decoding.
Identify the number of query heads (H_q), key/value heads (H_kv), head dimension (d), batch size (B), and current cache length (t). Note that H_q is a multiple of H_kv, and each KV head is shared by H_q/H_kv query heads.
For the new token, compute the query, key, and value vectors. The query has shape (B, 1, H_q, d). The key and value have shape (B, 1, H_kv, d) because they are shared across query head groups.
Append the new key and value to the cached K and V along the sequence dimension. The cache shapes change from (B, t, H_kv, d) to (B, t+1, H_kv, d).
For each query head, use its corresponding KV head (repeated H_q/H_kv times) to compute attention scores against the updated cache. The output shape is (B, 1, H_q, d).
State that the KV cache grows by one along the sequence dimension, while the output shape remains (B, 1, H_q, d). The cache shapes are (B, t+1, H_kv, d) for both K and V.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.