← Microsoft Interview Insights
Start by clarifying the scope and assumptions (e.g., batch-first vs sequence-first, masking requirements). Then implement each component modularly, explaining the math and shape transformations as you go. Finally, test with small examples to verify correctness, especially masking and weight tying.
Pro tip: Mention that you would use `torch.nn.functional.scaled_dot_product_attention` for efficiency in production, but implement from scratch here to demonstrate understanding. Also, highlight the importance of numerical stability (e.g., subtracting max before softmax) and proper initialization (Xavier/Glorot) for training stability.
Ask about input shapes, masking needs (padding, causal), and whether to include dropout. Define tensor dimensions (batch, seq_len, d_model, num_heads) and initialize parameters with correct shapes.
Compute Q, K, V projections, scale by sqrt(d_k), apply mask (if any) with -inf before softmax, and compute weighted sum. Explain the shape transformations and numerical stability trick.
Split Q, K, V into multiple heads, apply attention in parallel, concatenate heads, and project output. Discuss how to handle masking across heads and the role of the output projection.
Build the FFN with two linear layers and activation (e.g., ReLU/GELU), then assemble encoder and decoder layers with residual connections, layer norm, and dropout. For decoder, include masked self-attention and cross-attention.
Tie embedding and output projection weights if applicable, and run a forward pass with dummy data to check all shapes and masking behavior. Discuss parameter count and initialization.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Arithmetic intensity is just FLOPs divided by bytes moved, and you compare that to the machine's ridge point to figure out which bound you're hitting.
Start by defining the Roofline model and the arithmetic intensity (AI) of the operation, then compute AI for the given matrix multiplication and compare it to the machine balance (peak FLOPs / peak bandwidth). Use this comparison to classify the operation as compute-bound or memory-bound and predict wall-clock time as max(FLOPs/peak_FLOPs, bytes/peak_bandwidth). Finally, explain how varying batch size, sequence length, or hidden dimension changes AI and shifts the operating point on the Roofline plot.
Pro tip: Emphasize that real-world performance often falls short of the Roofline prediction due to factors like memory latency, cache effects, and kernel inefficiencies; mentioning these shows practical maturity beyond textbook theory.
Briefly explain the Roofline model: a log-log plot of attainable performance vs. arithmetic intensity, bounded by peak compute and peak memory bandwidth. The ridge point separates memory-bound and compute-bound regions.
For a matrix multiplication like QK^T (dimensions: batch B, heads H, sequence length S, head dim D), calculate FLOPs (2*B*H*S*S*D) and memory traffic (bytes for Q, K, and output). AI = FLOPs / bytes.
Calculate machine balance = peak FLOPs / peak memory bandwidth. If AI > machine balance, operation is compute-bound; else memory-bound. Predict wall-clock time as max(FLOPs/peak_FLOPs, bytes/peak_bandwidth).
Explain how changing batch size, sequence length, or hidden dimension affects AI. For example, increasing batch size or hidden dimension typically increases AI (more compute per byte), shifting toward compute-bound; increasing sequence length may have mixed effects depending on memory access patterns.
Mention that real-world performance may deviate from Roofline predictions due to memory hierarchy, kernel launch overhead, and parallelism. Suggest optimizations like tiling, fusion, or using tensor cores based on the bound.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.