Google·Machine Learning Engineer·Onsite - Coding / Algorithms
- Implement a multi-head self-attention module in Python (NumPy or PyTorch), taking an input of shape (batch_size, sequence_length, hidden_dim), splitting into multiple heads, computing scaled dot-product attention with optional masking, and projecting the concatenated output.
- Implement a SwiGLU feed-forward layer where the gate and value projections run in parallel, apply Swish activation to the gate path, multiply elementwise, then project the result.
- Put the attention and feed-forward components together into a full Transformer block with residual connections and layer normalization, returning output with the same shape as the input.
- Explain the time and memory complexity of multi-head attention, and walk through how masking works across the attention heads.
“The shape tracking is where things get slippery.”