I got the O(N² · d) time part right away but fumbled explaining why memory is O(N²) specifically.
Start by defining the self-attention operation and its inputs: queries, keys, and values matrices of shape N x d. Then derive the time and memory complexity by analyzing the matrix multiplications and the attention weight matrix. Finally, discuss the implications and potential optimizations.
Pro tip: Mention that while the standard self-attention has O(N^2 d) time and O(N^2) memory, many efficient variants reduce this to O(N d^2) or O(N log N) by approximating the attention matrix, which is crucial for long sequences.
Explain that self-attention computes attention scores as QK^T, applies softmax, and multiplies by V. Q, K, V are linear projections of the input, each of shape N x d.
Break down the matrix multiplications: QK^T takes O(N^2 d), softmax takes O(N^2), and multiplying by V takes O(N^2 d). Total time is O(N^2 d).
The attention weight matrix (N x N) dominates memory, requiring O(N^2) storage. Additionally, storing Q, K, V takes O(N d), but for large N, O(N^2) is the bottleneck.
Highlight that quadratic complexity limits sequence length, and mention common optimizations like sparse attention, low-rank approximations, or kernel methods that reduce complexity.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.