This is the kind of question where you think you know it until you're actually talking and realize you're fuzzy on the numbers.
Start by framing the GPU memory hierarchy as a trade-off between capacity, latency, bandwidth, and programmer control, then walk through each level from fastest/smallest to slowest/largest. For each level, explicitly state latency, bandwidth, capacity, and how much control the programmer has, and tie it back to performance implications like coalescing and occupancy.
Pro tip: Emphasize that shared memory and registers are explicitly managed by the programmer, while L1/L2 caches are hardware-managed but can be influenced via access patterns and cache hints. Mention that on modern NVIDIA GPUs (Volta+), L1 and shared memory are physically unified, which affects tuning.
Explain that GPU memory hierarchy is designed for massive parallelism, with each level trading off speed, size, and control. Mention that understanding this hierarchy is key to writing high-performance CUDA kernels.
Describe registers as the fastest, smallest storage, private to each thread, with latency ~1 cycle and bandwidth effectively unlimited per thread. Programmer controls allocation via variable declarations, but excessive use reduces occupancy.
Explain that shared memory is on-chip, low-latency (~20-30 cycles), high-bandwidth, and explicitly managed by the programmer for inter-thread communication. L1 cache is hardware-managed, similar latency, and on Volta+ shares physical space with shared memory.
Describe L2 as a larger, slower cache (~200 cycles latency) shared across the GPU, with higher bandwidth than HBM but lower than L1. Programmer has limited control, but access patterns and cache hints can influence hit rates.
Explain that HBM is off-chip, with the largest capacity (several GB to tens of GB), highest latency (~400-800 cycles), and lowest bandwidth relative to on-chip memory. Programmer controls allocation and access patterns; coalesced accesses are critical for performance.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Start by contrasting the massive compute throughput of GPUs with the limited bandwidth and capacity of off-chip memory, then explain how naive GEMM becomes memory-bound due to repeated global memory accesses. Introduce tiling as a technique to exploit data reuse in on-chip memory (shared memory/registers), reducing global memory traffic and increasing arithmetic intensity.
Pro tip: Quantify the impact: mention that tiling can reduce global memory accesses by a factor of the tile size, turning a memory-bound kernel into a compute-bound one. Also, note that tile size must balance shared memory usage, register pressure, and occupancy.
Outline the levels: global memory (large, high latency, low bandwidth), shared memory/L1 (small, low latency, high bandwidth), and registers (fastest, limited). Emphasize the bandwidth and latency gaps.
Explain that naive GEMM loads each element of A and B from global memory multiple times (O(N^3) loads for O(N^3) compute), leading to memory-bound performance because global memory bandwidth is the bottleneck.
Describe how tiling partitions the matrices into smaller blocks that fit in shared memory. Each block is loaded once from global memory and reused multiple times, reducing global memory traffic by a factor of the tile size.
Highlight increased arithmetic intensity, better latency hiding, and higher effective bandwidth. Discuss trade-offs: larger tiles reduce global traffic but may limit occupancy due to shared memory/register constraints.
Mention that libraries like cuBLAS and CUTLASS use tiling (and further optimizations like double buffering) to achieve near-peak performance on NVIDIA GPUs.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Knew the punchline: attention scores are O(n^2) in sequence length and writing them all to HBM then reading them back is the bottleneck, not compute.
Start by explaining the memory hierarchy and why HBM bandwidth is a bottleneck for attention. Then describe how FlashAttention's tiling and recomputation avoid materializing the full attention matrix, reducing HBM traffic. Finally, connect this to performance gains and trade-offs.
Pro tip: Quantify the impact: e.g., 'FlashAttention reduces HBM accesses by up to 10x, leading to 2-4x speedups on long sequences.' This shows you understand the practical significance.
Describe the GPU memory hierarchy: registers, shared memory/L1, L2, and HBM. Highlight the bandwidth and latency differences, and why HBM is the bottleneck for memory-bound operations.
Explain that standard attention computes and stores the full N×N attention matrix in HBM, causing O(N²) memory traffic and limiting sequence length.
Explain tiling: compute attention in blocks, keeping intermediate results in on-chip memory (shared memory/registers), and recompute in the backward pass instead of storing the matrix.
Discuss how this reduces HBM traffic, improves speed and memory efficiency, and enables longer sequences. Mention the trade-off of extra computation (recomputation) for less memory movement.
Relate to NVIDIA GPU specifics: e.g., tensor cores, shared memory capacity, and how FlashAttention is optimized for these architectures.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.
Arithmetic intensity is FLOPs per byte of memory traffic.
Start by defining arithmetic intensity as the ratio of floating-point operations to bytes of memory traffic, then explain how the roofline model plots this against peak compute and memory bandwidth to identify the limiting factor. Use the model to compare a kernel's operational intensity to the machine balance point, concluding whether it's compute-bound or memory-bound.
Pro tip: Mention that real-world kernels often have lower arithmetic intensity than theoretical due to memory access patterns, so profiling with tools like Nsight Compute is essential to validate roofline predictions.
Explain that arithmetic intensity is the number of floating-point operations (FLOPs) performed per byte of memory traffic (FLOPs/byte). It quantifies how much computation is done relative to data movement.
Describe the roofline model as a log-log plot with arithmetic intensity on the x-axis and achievable performance (FLOP/s) on the y-axis. It shows two ceilings: a slanted memory bandwidth roof and a flat compute roof.
The ridge point where the two roofs intersect represents the machine balance: peak compute divided by peak memory bandwidth. Kernels with intensity below this point are memory-bound; above, compute-bound.
Calculate or estimate the kernel's arithmetic intensity, then locate it on the roofline. If it falls under the slanted roof, it's memory-bound; if under the flat roof, it's compute-bound.
Explain that memory-bound kernels benefit from reducing memory traffic (e.g., caching, fusion), while compute-bound kernels benefit from increasing parallelism or using faster math instructions.
AI-generated suggestions, not part of the candidate's original notes. May be inaccurate — verify before relying on them.