I knew this but fumbled the explanation a bit.
Start by defining the core architectural difference: encoders use bidirectional self-attention, while decoders use causal (masked) self-attention plus cross-attention. Then explain how the causal mask enforces autoregressive generation and prevents information leakage, and discuss the trade-offs in terms of parallelism and training efficiency.
Pro tip: Emphasize that the causal mask is not just a technical detail but a fundamental design choice that enables autoregressive generation while maintaining parallel training. Mention that in practice, decoder-only models (like GPT) have become dominant for generation tasks, but encoder-decoder architectures (like T5) still excel in sequence-to-sequence tasks.
Explain that the encoder processes the entire input sequence bidirectionally to create rich representations, while the decoder generates the output sequence one token at a time, conditioning on previously generated tokens.
Detail that encoder self-attention allows each position to attend to all positions in the input. Decoder self-attention uses a causal mask to prevent attending to future positions, and decoder cross-attention attends to the encoder's output.
Discuss how the causal mask is a lower-triangular matrix of zeros and negative infinities that ensures position i can only attend to positions ≤ i. This enforces autoregressive property and prevents information leakage during training.
Compare the parallelism: encoders can process all positions in parallel, while decoders during training can also be parallelized due to masking, but during inference they are sequential. Mention that masking adds computational overhead but is essential for generation.
Give examples: BERT (encoder-only), GPT (decoder-only), T5 (encoder-decoder). Explain how the choice depends on the task: understanding vs. generation vs. sequence-to-sequence.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Start by explaining the role of stochastic sampling in autoregressive generation, then detail the conditions for determinism. Emphasize that determinism requires both greedy decoding and control over all sources of randomness, including hardware and software factors.
Pro tip: Mention that even with greedy decoding, non-determinism can arise from floating-point non-associativity and parallel operations, so true determinism often requires setting specific environment variables and using deterministic algorithms.
Describe how autoregressive models generate text by sampling from a probability distribution at each step, and how techniques like temperature, top-k, and top-p introduce randomness.
List the sources: random seed initialization, dropout (if active), and non-deterministic GPU operations (e.g., cuDNN autotuning, atomic operations).
Explain that greedy decoding (always picking the highest probability token) or beam search with a fixed seed can produce deterministic outputs if all randomness is controlled.
Mention that even with greedy decoding, floating-point precision, parallel reductions, and library versions can cause slight variations, so determinism requires setting seeds, disabling non-deterministic ops, and using deterministic algorithms.
Conclude that determinism is achievable when using greedy decoding, fixed random seeds, deterministic hardware/software settings, and no dropout or other stochastic layers.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.