I could write out softmax(QK^T / sqrt(d_k))V fine but stumbled explaining the scaling part.
Start by defining the attention mechanism and its components (queries, keys, values), then derive the scaled dot-product attention formula step by step. Explain the variance argument for scaling by 1/√d_k, and discuss the practical implications for training stability and performance.
Pro tip: Mention that while scaling is crucial for large d_k, some modern architectures use alternative scaling or normalization techniques; showing awareness of these variations demonstrates depth.
Introduce queries (Q), keys (K), and values (V) as learned linear projections of input representations. Explain that attention computes a weighted sum of values based on compatibility between queries and keys.
Compute compatibility scores as dot products between queries and keys: S = QK^T. Apply softmax to get attention weights: A = softmax(S). The output is A V.
Show that if Q and K have independent components with zero mean and unit variance, the dot product q·k has variance d_k. For large d_k, the scores become large in magnitude, pushing softmax into saturated regions with tiny gradients.
Divide the dot products by √d_k to normalize the variance to 1. This keeps the softmax inputs in a reasonable range, preventing vanishing gradients and stabilizing training.
Explain that scaling allows for stable training with larger d_k, which is common in Transformer models. Mention that without scaling, models may fail to converge or require careful initialization and learning rate tuning.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Covered multi-head attention, residual connections, LayerNorm, and the FFN without issue.
Start by walking through the transformer block components in order: multi-head self-attention, feed-forward network, residual connections, and layer normalization. Then explain pre-norm vs post-norm by contrasting where layer normalization is placed relative to the sublayers, and discuss the implications for training stability and performance.
Pro tip: Mention that pre-norm is standard in modern large language models like GPT because it enables stable training without warmup, while post-norm, though original, often requires careful learning rate warmup and can be less stable.
Explain that a transformer block contains a multi-head self-attention mechanism and a position-wise feed-forward network, each followed by residual connections and layer normalization.
Break down multi-head self-attention: linear projections to queries, keys, values; scaled dot-product attention; concatenation of heads; and output projection.
Describe the FFN as two linear transformations with a ReLU activation in between, typically expanding and then projecting back to the model dimension.
Discuss how residual connections add the input to the sublayer output, and how layer normalization normalizes activations across the feature dimension.
Explain that in pre-norm, layer normalization is applied before each sublayer (LN -> Sublayer -> Residual), while in post-norm, it is applied after the residual addition (Sublayer -> Residual -> LN). Discuss trade-offs: pre-norm improves gradient flow and training stability, especially for deep models, while post-norm can yield better performance in some settings but requires careful initialization and warmup.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Start by defining causal and bidirectional attention masking, then compare their computational and representational trade-offs. Finally, map each to its primary use cases, emphasizing why the choice matters for different tasks and architectures.
Pro tip: Mention that causal masking is essential for autoregressive generation to prevent information leakage, while bidirectional masking is used for understanding tasks where full context is available. Also note that some models like T5 use a mix (e.g., encoder bidirectional, decoder causal).
Explain that causal masking allows each position to attend only to previous positions, while bidirectional masking allows attention to all positions.
Discuss how causal masking enforces a triangular attention matrix, which is efficient for sequential generation, while bidirectional masking uses a full matrix, capturing richer context but requiring full sequence input.
State that causal masking is used in autoregressive language models (e.g., GPT) for text generation, while bidirectional masking is used in masked language models (e.g., BERT) for understanding tasks like classification.
Mention that encoder-decoder models (e.g., T5) use bidirectional masking in the encoder and causal masking in the decoder, combining both for sequence-to-sequence tasks.
Summarize that causal masking enables efficient generation but limits context, while bidirectional masking provides full context but cannot be used for autoregressive generation.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Start by categorizing positional encoding schemes into absolute (sinusoidal, learned) and relative (bias, rotary), then compare their mechanisms and tradeoffs. Focus on extrapolation to longer sequences, discussing how each scheme generalizes beyond training length and the implications for model performance.
Pro tip: Emphasize that rotary encodings (RoPE) have become the de facto standard for long-context LLMs due to their relative nature and efficient extrapolation, but mention that techniques like YaRN or linear scaling can further enhance them. This shows awareness of current industry practices.
Briefly classify positional encodings into absolute (sinusoidal, learned) and relative (relative bias, rotary). This sets a clear structure for comparison.
For each type, describe how it encodes position: sinusoidal uses fixed sinusoids, learned uses trainable embeddings, relative bias adds learned biases based on distance, and rotary applies rotation to query/key vectors.
Discuss computational cost, parameter efficiency, and compatibility with attention mechanisms. For example, learned absolute requires fixed max length, while relative methods handle variable lengths better.
Evaluate how each scheme performs when sequence length exceeds training length. Sinusoidal can extrapolate but may degrade; learned absolute fails; relative bias and rotary extrapolate better, with rotary showing strong performance.
Summarize which schemes are best for long-context applications, mentioning that rotary is widely adopted but may need scaling techniques for extreme lengths.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Inverted dropout scales survivors by 1/(1-p) during training so you don't have to do anything special at inference time.
Start by contrasting dropout's stochastic behavior during training with its deterministic identity mapping at inference. Then explain the scaling factor (1/(1-p)) as a way to keep expected activations consistent between the two phases, avoiding a train-test mismatch.
Pro tip: Mention that some frameworks use inverted dropout (scaling during training) while others scale at inference; knowing this distinction shows practical experience and awareness of implementation details.
Explain that during training, each neuron is independently dropped with probability p, and surviving activations are scaled by 1/(1-p) to maintain the expected sum.
State that at inference, dropout is turned off: all neurons are active and no random dropping occurs, effectively using the full network.
Justify the scaling: without it, the expected output at inference would be larger by a factor of 1/(1-p) compared to training, causing a shift in activation distributions.
Note that the common implementation scales during training (inverted dropout), so inference requires no modification, simplifying deployment.
Conclude that this design ensures consistent expected activations, reduces overfitting, and maintains a single inference path without stochasticity.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.