Start by defining the core architectural difference: encoders process the entire input bidirectionally to build rich representations, while decoders generate outputs autoregressively using causal masking. Then map each to its typical use cases—encoders for understanding tasks like classification, decoders for generation tasks like text synthesis—and briefly mention encoder-decoder hybrids for sequence-to-sequence problems.
Pro tip: Tie the architecture choice to concrete trade-offs in latency, memory, and task suitability—e.g., encoders can parallelize but can't generate, decoders generate but are slower—and mention how this influences system design decisions at scale.
Explain that encoders use bidirectional self-attention (no masking) to see the full input, while decoders use causal (masked) self-attention to prevent looking ahead during generation.
Highlight that encoders are typically trained with masked language modeling or similar objectives to learn representations, whereas decoders are trained with next-token prediction to model a probability distribution over sequences.
Give examples: encoders for understanding tasks (classification, NER, sentiment analysis), decoders for generation tasks (text completion, summarization, translation), and encoder-decoder models for sequence-to-sequence tasks (translation, summarization).
Compare inference speed, memory usage, and flexibility: encoders can process inputs in parallel but cannot generate; decoders generate sequentially but are slower; encoder-decoder models combine both but add complexity.
Mention popular models (BERT as encoder, GPT as decoder, T5 as encoder-decoder) and relate to Adobe's use cases like document understanding, image captioning, or content generation.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
This is the kind of question that sounds easy until you're actually explaining it out loud.
Start by defining both attention mechanisms clearly, then contrast their information flow and training objectives. Explain why each exists by linking to specific architectures and tasks, such as bidirectional attention for understanding tasks and causal attention for generation.
Pro tip: Mention that causal masking is essential for autoregressive generation to prevent information leakage, while bidirectional attention enables richer context for tasks like classification or question answering. Also note that some models combine both, e.g., prefix LM or encoder-decoder architectures.
Explain that in bidirectional attention, each token can attend to all other tokens in the sequence, both left and right. This allows the model to build a deep, context-aware representation of each token.
Explain that causal masked attention restricts each token to attend only to previous tokens and itself, preventing access to future tokens. This is achieved via a triangular mask.
Highlight that bidirectional attention has full information flow, while causal attention enforces a unidirectional flow. This difference impacts how representations are formed and what tasks each is suited for.
Discuss that bidirectional attention is used in models like BERT for tasks requiring full context (e.g., classification, QA). Causal attention is used in autoregressive models like GPT for generation, where future tokens must not be seen during training.
Summarize that bidirectional attention captures richer context but cannot be used for generation without leakage, while causal attention enables generation but may have less context. Some architectures combine both (e.g., encoder-decoder, prefix LM).
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Managed to explain that the decoder generates queries while keys and values come from the encoder output.
Start by defining cross-attention as the mechanism that lets the decoder attend to encoder outputs, then explain the query-key-value computation and how it differs from self-attention. Finally, connect it to the encoder-decoder workflow and discuss trade-offs like computational cost and design choices.
Pro tip: Emphasize that cross-attention is the bridge between modalities or languages, and mention that its efficiency can be optimized via techniques like caching or sparse attention—showing you think beyond textbook definitions.
Explain that cross-attention allows the decoder to focus on relevant parts of the encoder's output by computing attention between decoder queries and encoder keys/values.
Walk through the steps: project decoder hidden states to queries, encoder outputs to keys and values, compute scaled dot-product attention, and produce a weighted sum of values.
Highlight that in self-attention, queries, keys, and values come from the same sequence, while in cross-attention they come from different sequences (decoder vs. encoder).
Discuss how cross-attention is inserted between self-attention and feed-forward layers in each decoder block, enabling the decoder to condition on the entire input sequence.
Mention computational complexity (O(n*m) for sequence lengths n and m), memory usage, and techniques like caching encoder outputs or using sparse attention for efficiency.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
This one tripped me up more than it should have.
Start by clarifying the fundamental difference in how encoder-only and decoder-only models use attention during inference. Explain that encoder-only models process the entire input sequence in parallel with bidirectional attention, so there is no need to cache past keys/values because each token's representation is computed once and reused. Contrast this with autoregressive decoding, where the KV cache is essential to avoid recomputing past tokens.
Pro tip: Emphasize that the KV cache is an optimization for autoregressive generation, not a requirement for attention itself; encoder-only models are not generative in the same way, so the concept doesn't apply. This shows you understand the underlying mechanics rather than just memorizing facts.
Explain that encoder-only models (e.g., BERT) take a full input sequence and produce contextualized embeddings for each token in a single forward pass, with no autoregressive generation.
Highlight that self-attention is bidirectional and computed over the entire sequence at once, so all keys and values are available simultaneously and used only once per inference.
Clarify that in autoregressive decoding, tokens are generated one at a time, and the KV cache stores previously computed keys and values to avoid recomputing them for each new token.
Point out that encoder-only models do not generate tokens sequentially, so there is no repeated use of past keys/values; hence a cache would provide no benefit and is unnecessary.
Summarize that the absence of a KV cache in encoder-only inference is a natural consequence of their parallel, non-autoregressive nature, and mention that this makes them efficient for tasks like classification or embedding extraction.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.