Start by explaining why positional encoding is needed in Transformers, then describe the original sinusoidal method and its properties. Next, cover modern alternatives like learned embeddings, relative position, and rotary embeddings, and finally discuss how each choice impacts extrapolation to longer contexts, tying it to practical trade-offs.
Pro tip: Emphasize that the choice of positional encoding is a trade-off between simplicity, performance, and length generalization; mention that rotary embeddings (RoPE) are currently favored for long-context models due to their relative nature and efficient extrapolation.
Describe why Transformers need positional information: self-attention is permutation-invariant, so without positional encoding, the model cannot distinguish order or distance between tokens.
Explain the original sinusoidal approach: fixed, deterministic functions using sine and cosine of different frequencies, added to input embeddings. Mention its ability to represent relative positions via linear relationships.
Discuss learned positional embeddings, relative position representations (e.g., Shaw et al., T5), and rotary position embeddings (RoPE). Highlight their motivations and key differences.
Compare how each method handles sequences longer than those seen during training: sinusoidal can extrapolate somewhat but degrades; learned embeddings cannot extrapolate; relative and rotary methods offer better extrapolation, with RoPE being particularly effective.
Conclude with practical considerations: sinusoidal is simple but limited; learned is flexible but not length-general; relative and rotary improve long-context performance at the cost of complexity.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.