Scale.ai·Machine Learning Engineer·Technical Phone Screen·Senior

Senior

Apr 2026

Summary

Scale.ai ML Engineer technical screen, pretty deep on transformer internals. The whole session was basically one long implementation question with a lot of follow-up threads pulling off of it.

Questions Asked (1)

Implement multi-head self-attention from scratch using NumPy or PyTorch. Given Q, K, V projections and a number of heads, compute scaled dot-product attention per head, concatenate the results, and pass through an output projection. Also explain causal and padding masking, and why you scale by 1/sqrt(d_k).

System DesignTechnical Trade-offsAlgorithms & Data Structures

Author's notes

I started with the math and worked forward, splitting Q/K/V into heads by reshaping, running softmax(QK^T / sqrt(d_k))V per head, then cat and project.

Create a free account to read the full note

AI HintsAI Generated

Suggested Approach

Start by clarifying the input shapes and the desired output, then implement the attention mechanism step-by-step: linear projections, splitting into heads, scaled dot-product attention with optional masking, concatenation, and output projection. Explain the purpose of scaling and masking, and discuss trade-offs like computational complexity and memory usage.

Pro tip: Mention that you would use `torch.einsum` or `torch.matmul` with broadcasting for efficient batched computation, and that you would verify correctness by comparing against a reference implementation like PyTorch's `nn.MultiheadAttention`.

1. Clarify inputs and outputs

Confirm the shapes of Q, K, V (batch_size, seq_len, d_model) and the number of heads. Determine the output shape and whether masking is needed.

2. Implement linear projections and head splitting

Apply learned linear projections to Q, K, V to get d_model dimensions, then reshape and transpose to split into heads: (batch_size, num_heads, seq_len, d_k).

3. Compute scaled dot-product attention

Compute attention scores as Q @ K^T / sqrt(d_k), apply optional masks (causal or padding), apply softmax, and multiply by V to get per-head outputs.

4. Concatenate heads and apply output projection

Transpose and reshape the per-head outputs back to (batch_size, seq_len, d_model), then apply a final linear projection.

5. Explain masking and scaling

Describe causal masking (preventing attention to future tokens) and padding masking (ignoring padding tokens), and justify scaling by 1/sqrt(d_k) to stabilize gradients.

Key Points to Mention

Scaling by 1/sqrt(d_k) prevents softmax saturation and vanishing gradients when d_k is large.
Causal masking ensures autoregressive property by setting future positions to -inf before softmax.
Padding masking ignores padding tokens by setting their attention scores to -inf.
Multi-head attention allows the model to attend to information from different representation subspaces.
Computational complexity is O(n^2 * d) for sequence length n, which can be optimized with efficient attention variants.
Implementation details: use of einsum or matmul for batched operations, and proper reshaping/transposing.

AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.